MEMORY OVERCOMMIT SETTINGS

To see your memory system now, under ‘default’ settings, enter the following into terminal:

sudo cat /proc/meminfo

We can see lots of lines but the four we’re interested in are:

MemTotal: The total amount of physical RAM available on your system.

MemFree: The total amount of physical RAM not being used for anything.

CommitLimit: The total amount of memory, both RAM and SWAP, available to commit to the running and requested applications (not necessarily directly related to the actual physical RAM amount, we will see why later).

Commited_AS: The total amount of memory required in the worse case scenario right now if all the applications actually used what they asked for at startup!

If the application/s needed what they originally asked for, an out-of-memory or ‘OOM’ would happen. This would mean that the OOM-killer would kick in to try and free up actual memory by killing running processes it thinks might help to free up memory. By then though a kernel-panic (or at best  X11 would hang) might have happened resulting in a frozen system (aka blue-screen in MS terms) or of course OOM-killer killed a critical system process.

To solve the random selections of the OOM-killer potentially killing off a critical system process, or not kicking in prior to a kernel-panic, we can change the following:

vm.overcommit_ratio=100: The percentage of total actual memory resources available to applications. This might be the total of RAM + SWAP, or just RAM if you have no SWAP. (IE: RAM=1gb & SWAP=1gb, overcommit_ratio=100 would mean 2gb could be allocated to applications. overcommit_ratio=50 would mean 1gb could be allocated to applications – this would obviously not be a sensible choice as 1gb would never be used!)

vm.overcommit_memory=2: This tells the kernel to never agree to allocate more than the total percentage of actual memory determined by overcommit_ratio= and disables the OOM-killer daemon.

We can change the above settings by entering the following into terminal:

sudo sync    — this tells any files in cache on RAM to write to disk now

sudo sh -c “sync; echo 3 > /proc/sys/vm/drop_caches”    — this drops all caches from RAM

sudo cat /proc/meminfo    — check that Committed_AS is below CommitLimit

sudo sysctl -w vm.overcommit_ratio=99    — use 99% of physical memory

sudo sysctl -w vm.overcommit_memory=2    — only allow applications to start if there is enough memory determined by the above command

So now when we try to open a memory hungry application, or we have to many applications open already, the new application is refused with a notification that IE: ‘file manager failed to fork’, or failed to start because there isn’t the available memory. Potentially the application could theoretically start with what memory is available now, but it may continue to require memory to a point the system is unusable as a result and hangs or crashes. A web-browser would be a good example, it opens with only one tab, but during the day you open a dozen more, at some point memory would be exhausted.

By using the two above tweaks we end up with a system that cannot agree to give applications more memory allocation than it physically has. This stops hangs or kernel panics that render the entire system useless, potentially losing those last bits of information you were inputting, instead it simply tells you that there is no more memory, you need to go buy more RAM!

We now know our system will just tell us there’s no more memory for that new application to open, and we like it, we want these settings to survive power cycles (rebooting), we do this by adding the above commands into:

sudo gedit /etc/sysctl.conf    — I use gedit, but nano, vi etc all work

Add: sudo sysctl -w vm.overcommit_ratio=99 and sudo sysctl -w vm.overcommit_memory=2 to the bottom of that file on separate lines and save. Mine look like this:

#system tweaks
vm.swappiness=5
vm.vfs_cache_pressure=50
vm.overcommit_ratio=99
vm.overcommit_memory=2

(I use 99% just to give a little allowance).

Of course you could increase the size of your SWAP partition as CommitLimit is a total of RAM+SWAP (remembering that SWAP is disk based so slower than RAM) so you can open all those tabs, or applications without getting ‘failed to fork’ messages, or you could add a SWAP partition if you haven’t got one already.

“But I have an SSD and SWAP is bad”, well yes it is if you are constantly using it because you only have 1gb of RAM! If you have 4+gb of ram, and depending on what you use your system for, SWAP on an SSD would act as a final safety net saving you from kernel panic under stock settings, or by using the above settings it would stop the constant ‘failed to fork’, but if that’s a regular message following these changes i’d suggest you buy more RAM!

NB: Default is: vm.overcommit_memory=0 which means in short that no tabs are kept on actual available memory space, the kernel agrees to all requests for memory from applications and OOM-killer is activated, in my experience followed by hangs and reboots

kernel:NMI watchdog: BUG: soft lockup – CPU#0 stuck for 21s!

  • Consulting the files /etc/grub.conf and /boot/grub/grub.conf, in RHEL 6 and below, or /etc/sysconfig/grub in RHEL 7, it should be verified if the console output is redirected to a console, i.e. using console=ttyS1 or console=ttyS1,9600. In both of these cases the output is restricted to 9600 baud, limiting the output and possibly causing issues.
  • A fix might be to not log to the serial console, or explicitly configure a higher baudrate, i.e. using console=ttyS1,115200. Please note, in some situations also 115200 baud might be a limiting factor.

Otherwise, investigate further root cause conditions

  • Determine if the system was under extremely high load at the time the soft lockups were seen in the logs. If the sysstat package was already installed, it will have recorded load average every 10 minutes using a cron job.
  • Then Load average can be found by searching for ldavg in /var/log/sa/sar<day> where day is the number date of the day when soft lockups were seen. If load average is significantly higher than the amount of logical CPU cores on the system it indicates the soft lockups probably occured because of extremely high workloads.
    In this case it would be best to determine what processes caused the load to go so high and make changes so that the processes don’t cause the issue again.
  • Since it is also possible that defects in the kernel could have caused the soft lockups, full logs needs to be investigated around the time of the soft lockups to see if the issue is a bug or is fixed by errata. It can help to look in the changelog of the latest kernel available on Red Hat Network and see if any soft lockup issues were fixed since the version of the installed kernel.
  • Another way is to eliminate the possibility of a known issue which has already been fixed by testing the system by running it with the latest kernel and see if the soft lockups happen again. Red Hat support may be required to conclusively determine if the issue is a bug.
  • Also verify with a hardware vendor that the issue is not hardware related. One way to verify that the issue is not a known and solved hardware problem is to update the firmware or BIOS to the latest available from the hardware vendor.
  • On virtual systems, soft lockups can indicate that the underlying hypervisor is overcommitted. Please see this article addressing this issue: VMware virtual machine guest suffers multiple soft lockups at the same time
  • If all of the above have been verified to not be the cause it could be a case where soft lockups do not indicate a problem; for example on systems with very large numbers of CPU cores.

If this is encountered in RHEL 5, then increase the threshold at which the messages appear using the following procedures:

  • Run following command and check whether “soft lockup” errors are still encountered on the system:
    # sysctl -w kernel.softlockup_thresh=30
  • To make this parameter persistent across reboots by adding following line in /etc/sysctl.conf file:
     kernel.softlockup_thresh=30

In RHEL 6 and above, the threshold is now named “watchdog_thresh” and can be set to no higher than 60:
– To make this change in RHEL 6 and above, set the tuneable kernel.watchdog_thresh in sysctl.conf

Additional Notes:

  • The softlockup_thresh kernel parameter was introduced in Red Hat Enterprise Linux 5.2 in kernel-2.6.18-92.el5 thus it is not possible to modify this on older versions.

Root Cause

  • Soft lockups are situations in which the kernel’s scheduler subsystem has not been given a chance to perform its job for more than the limit set by the watchdog threshold, in seconds; they can be caused by defects in the kernel, by hardware issues or by extremely high workloads.
  • If lockups are encountered on a virtual system, it is important to ensure that the hypervisor is not overcommitted.
  • Hardware issues related to newly installed memory might cause soft lockups.
  • Also misconfigurations might cause the issue, like redirecting console output to a serial device and limiting it to i.e. 9600 baud.
  • On systems with a very large numbers of CPU cores soft lockups might not indicate a problem.