When IO threads are running, they use busy-wait loop to wait for jobs. Therefore, they run at 100% all the time and CPU utilization is not a useful metric to understand the load.
When I/O threads are used, the main thread does non-blocking epoll() so it too does busy-looping and runs at 100% CPU. Therefore we need a CPU utilization metric for the main thread too.
The problem is explained in this issue:
Proposed INFO fields
Measure the "active" running time = wall clock minus the time spent on busy-waiting and blocking epoll.
# CPU
used_cpu_sys:68.476389
used_cpu_user:219.408941
used_cpu_sys_children:0.003157
used_cpu_user_children:0.008264
used_cpu_sys_main_thread:68.478353
used_cpu_user_main_thread:219.404508
+used_active_time_main_thread:123.456789
+used_active_time_io_thread_1:97.013234
+used_active_time_io_thread_2:45.678901
Implementation in two PRs:
Alternatives considered
Naming
- "used": The word "used" is already used for CPU time actually consumed by user space or kernel space in the INFO fields
used_cpu_sys_main_thread and used_cpu_user_main_thread.
- "useful": This word is a little similar to "used" so they can be confusing.
- "utilization": This word is a little long and also similar to "useful" and "used".
- "busy": This work is often used in words like busy-wait looping, which is the non-useful activity. It may be confusing to use this word for the useful time metric.
- "active": This is my current preference. We can talk about CPU utilization as "active CPU time" = "used CPU time" minus "busy-waiting".
Why not more fields for I/O threads?
Since I/O threads always run at 100% CPU, there's not much point in presenting the actual used CPU time.
Why measure active time using monotonic clock?
An alternative to measuring the duration of active work using a monotonic clock is to use CPU-time clock, clock_gettime(). However, a low CPU usage is no guarantee that the thread has spare capacity to handle more work. In addition to waiting for work, it may be preempted by the kernel or wait for blocking IO or other syscalls.
OTOH, measuring active time using a monotonic clock and it revelase that 90% of the time consists of active work, it implies that the remaining 10% of the time, the thread has been waiting for more work. This is inline with the purpose of these metrics: to monitor the spare capacity of the threads.
Why not present a percentage?
If we present percentage for each thread (e.g. utilization_io_thread_1:60%), we would need to compute it over some specified time window.
If we use an instantaneous metric (16 samples over the last 1.6 seconds) we can miss a CPU utilization spike if we call INFO every five minutes or so. Presenting the cumulative metric allows the user to choose the time window.
Do we need to compare the IO thread's active time to the IO thread's uptime?
For I/O thread's active time to be be useful and to calculate a percentage over a time window, it may be tempting to have the I/O thread's uptime, so the caller can compute a percentage and compare it to the uptime delta.
However, it is not strictly necessary, so I don't recommend adding it. Even without it, the caller can compare the the delta between two points in time and use this delta to compare the I/O thread's active time delta between two INFO calls. To calculate the exact time between two calls to INFO, the delta of the INFO field for the server's uptime can be used.
When IO threads are running, they use busy-wait loop to wait for jobs. Therefore, they run at 100% all the time and CPU utilization is not a useful metric to understand the load.
When I/O threads are used, the main thread does non-blocking
epoll()so it too does busy-looping and runs at 100% CPU. Therefore we need a CPU utilization metric for the main thread too.The problem is explained in this issue:
Proposed INFO fields
Measure the "active" running time = wall clock minus the time spent on busy-waiting and blocking epoll.
Implementation in two PRs:
Alternatives considered
Naming
used_cpu_sys_main_threadandused_cpu_user_main_thread.Why not more fields for I/O threads?
Since I/O threads always run at 100% CPU, there's not much point in presenting the actual used CPU time.
Why measure active time using monotonic clock?
An alternative to measuring the duration of active work using a monotonic clock is to use CPU-time clock,
clock_gettime(). However, a low CPU usage is no guarantee that the thread has spare capacity to handle more work. In addition to waiting for work, it may be preempted by the kernel or wait for blocking IO or other syscalls.OTOH, measuring active time using a monotonic clock and it revelase that 90% of the time consists of active work, it implies that the remaining 10% of the time, the thread has been waiting for more work. This is inline with the purpose of these metrics: to monitor the spare capacity of the threads.
Why not present a percentage?
If we present percentage for each thread (e.g.
utilization_io_thread_1:60%), we would need to compute it over some specified time window.If we use an instantaneous metric (16 samples over the last 1.6 seconds) we can miss a CPU utilization spike if we call INFO every five minutes or so. Presenting the cumulative metric allows the user to choose the time window.
Do we need to compare the IO thread's active time to the IO thread's uptime?
For I/O thread's active time to be be useful and to calculate a percentage over a time window, it may be tempting to have the I/O thread's uptime, so the caller can compute a percentage and compare it to the uptime delta.
However, it is not strictly necessary, so I don't recommend adding it. Even without it, the caller can compare the the delta between two points in time and use this delta to compare the I/O thread's active time delta between two INFO calls. To calculate the exact time between two calls to INFO, the delta of the INFO field for the server's uptime can be used.