Understanding processor usage and attributes is essential for optimizing the performance of Ubuntu servers. As a system resource, the CPU executes all programs and workloads, making it critical for overall responsiveness. This in-depth guide will cover CPU monitoring, analysis and tuning techniques for the administrator managing Linux systems.
CPU Architecture and Resources Refresher
Modern server grade processors from Intel and AMD provide extensive capabilities for demanding workloads. Key concepts that relate to Linux performance include:
Cores – Independent processing units that can handle threads in parallel. Most servers have 8-64+ physical cores.
Threads – Smallest sequence of instructions executed on a core. Hyperthreading provides 2 threads per core.
Sockets – Physical CPU packages plugged into the motherboard, often containing multiple cores.
Caches – Small local memory storing frequently accessed data, faster than main RAM access. L1, L2 and L3 caching is arranged in multiple levels per core.
Pipelines – Overlapping staged execution of individual instructions to improve throughput by working on multiple concurrently.
instruction sets – Low level binary encoding of operations the CPU can perform like x86-64, ARM and Power ISA. Newer extensions speed up specialized tasks.
Linux schedules running processes and threads across available cores and caches to maximize utilization of the underlying CPU resources through the scheduler and memory manager. Understanding this architecture helps optimize deploying applications onto the server. Next we will explore Ubuntu tools to analyze how the CPU is being used.
Top Usage Overview with Load Averages
The top utility provides the common starting point for checking on the health of Ubuntu server‘s CPU in real-time.
Here is an example output section:
top - 17:05:07 up 21:56, 2 users, load average: 0.50, 0.62, 0.59The load average stats indicate the CPU demand by processes over the last 1, 5 and 15 minute intervals. The expected threshold to stay under is 1.0 per CPU core on average. So for this 16 core server, load averages consistently over 16 would indicate potential performance issues.
Spikes over that limit suggests the server is CPU constrained for brief periods. Sustained higher load average flags a capacity shortage needing upgraded or additional servers. Very high load averages can cause visible system slowness or crashes.
Below the output also shows two active users and 99% idle CPU time so the current 0.5 load average is not yet problematic. Tuning efforts would start with the processes using the most CPU during the period of high demand…
Resource Constrained Process Identification with htop
While top shows the current most CPU intensive running processes, htop provides an enhanced interactive interface and metrics for identifying issues. Install htop:
sudo apt install htopLaunch htop and press F2 to setup a display with the most relevant columns for constrained resource analysis:
Sorting by CPU, Memory or Disk I/O usage quickly highlights processes to investigate for excessive usage. Common culprits include batch jobs, scripts with runaway loops, memory leaks, or just applications needing assignment to higher powered hardware. The various color coded bars help identify outliers in resource usage.
Additionally pressing F3 will hide kernel and background processes allowing focus just on user applications and services. F4 toggles displaying processes in a tree structure grouped by parent process ID. Combine sorting and tree view to troubleshoot by service such as just the Apache web server workload.
Htop reveals programs needing resource usage optimization or redeployment to alternative servers. Capturing detail for suspected poorly performing processes guides tuning.
Long Term Statistics with sar and sysstat
While top and htop show current server state, having longer term historical performance data enables identifying usage trends. The sysstat tools provide extensive statistics recorded over hours, days and weeks.
Install sysstat:
sudo apt install sysstatEnable automatic background data collection every 10 minutes with:
sudo systemctl enable sysstatNow activity metrics including CPU, memory, network, disks and processes will be captured to the /var/log/sysstat/ sa[0-9][0-9] files. No further setup is required.
The sar utility included in sysstat can report on the collected data with flexible filters and output. Some CPU focused examples:
# Hourly CPU usage past 24 hours
sar -u -f /var/log/sysstat/sa31 | grep -e "Average"
# Show CPU iowait % by day
sar -b -f /var/log/sysstat/sa25 | grep "iowait"
# 3 Second intervals for first 5 minutes for a specific date
sar -P ALL 3 5 -f /var/log/sysstat/sa25.07.2021 This makes visually identifying usage spikes on a particular date or times much easier. Also finding correlations between periods of high usage and technical issues or infrastructure changes is possible.
Sysstat collection should be enabled by default on all production Ubuntu servers. The data proves invaluable for both real-time and historical performance analysis.
Visualizing Data with GNOME System Monitor
When needing graphical CPU stats or convenience on a remote desktop session on the Ubuntu server, utilize the GNOME System Monitor application. It provides real-time per core usage graphs, filtering on metrics like memory or disk activity and process specific statistics.
Drilling into a process shows all resources consumed by open files, network sockets, child processes and more. The Sparklines graph at the bottom visualizes historical CPU demand for troubleshooting intermittent issues.
Combining GNOME System Monitor with the sysstat logged data in dashboards gives a robust graphical view into ongoing server performance suitable even for presentations and management.
FAQs on Linux CPU Performance
Q: Should I optimize for CPU cores or clock speed?
A: For parallelized workloads like web servers, databases and computational jobs, more cores allow higher throughput and responsiveness. Clock speed boosts single threaded program speed. A mix is idea, but scale-out with more cores first.
Q: When should I isolate CPUs for dedicated apps?
A: Shielding high priority latency sensitive processes from noise by containerization or CPU pinning helps for the most demanding workloads once proven necessary by benchmarks. Attempt optimization first.
Q: What causes high load averages when utilization looks low?
A: Symptoms like large load averages with 90%+ idle CPU time signals pressure from resource saturation in process scheduling and context switching between processes. Identify next constrained resource pushing CPU usage.
Q: How can CPU usage spike if limits are set?
A: Limits help contain processes but a burst of scheduled unlimited kernel threads, interrupts handlers or run away descheduled tasks can still strain the CPU. Enforce equilibrium across all activity through cgroups.
Expert Linux CPU Performance Tuning Techniques
Beyond monitoring and alerting on CPU usage anomalies, advanced tuning opportunities further optimize Ubuntu responsiveness. Applying techniques like isolation, binding, shielding, scheduling tuning, irq balancing and more squeeze out the most application throughput from the processors.
Some examples include:
Control Groups – Limit and isolate CPU cycles available to a processes hierarchy preventing excess usage
IRQ Balancing – Distribute device interrupt handling evenly across CPUs detecting imbalanced workload
CPU Pinning – Bind specific apps to dedicated cores/sockets reducing cache contention and jitter
CPU Governor Tuning – Switch to performance mode for throughput or conservative for power saving to reduce thermal output
Thread Concurrency Limiting – Prevent oversaturation and trashing of available cores through setting application thread/process pools
Batch Job Shielding – Prioritization and isolation of resources between critical and best effort workloads
FlameGraph Generation – Profile code execution visually to identify optimization improvement opportunities
There are always opportunities to boost performance through low level tuning as workloads evolve and increase over time on long running production Ubuntu servers.
Conclusion
Having visibility into CPU statistics is the first step for containment and optimization of growing application demands. Ubuntu provides versatile tools catering to various use cases of real-time monitoring, remote graphical analysis and extensive long term reporting. Performance dashboarding to detect and report anomalies is key for stability. Combine preventative resource management through cgroups with profiling tools targeting speed improvements in underlying programs. Record baseline metrics early on for comparison as upgrades occur. With a workflow integrating measurement capabilities, continuous improvement in responsiveness, speed and efficiency is achievable.
