The go runtime defaults to MADV_FREE by default. In that case pages are still assigned with the Beat, even after the pages have been "returned" to the OS. The kernel will eventually claim these pages, but RSS memory usage is reported as very high (which it is not necessarily). This seems to mess with the OOM killer at times.

As workaround one can tell the runtime to use MADV_DONTNEED via GODEBUG="madvdontneed=1". We should introduce this environment variable to the systemd file, init file, and docker file by default.

What to do

• Add GODEBUG="madvdontneed=1" to docker images.
• Add GODEBUG="madvdontneed=1" to systemd unit file.
• Add GODEBUG="madvdontneed=1" to init file.
• Add a note about this in known issues for 7.x maintained versions.
• Backport the changes to 7.10 too.

More information

• madvdontneed=1 will be the default in Go 1.16 https://go-review.googlesource.com/c/go/+/267100/
• https://blog.detectify.com/2019/09/05/how-we-tracked-down-a-memory-leak-in-one-of-our-go-microservices/