Memory leak on K8S #16183
Description
System information (version)
- OpenCV => 4.1.0.25
- Operating System / Platform => ubuntu 18.04 image on k8s (host: centos 7.6)
- Docker version 19.03.4, build 9013bf583a
- K8S v1.15.3
- Compiler => installed with python pip
Detailed description
I have a tornado web service container using cv2.imdecode, and it causes memory leaking when I deployed the image to k8s, but it would not happen on the same machine with native docker rather than k8s.
I can reproduce the problem with the below code.
I used gdb info threads to find out that there were a lot of threads not collected:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f78a8376580 (LWP 5991) "python" 0x00007f78a7c85603 in select () at ../sysdeps/unix/syscall-template.S:84
2 Thread 0x7f789ed93700 (LWP 5992) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
3 Thread 0x7f789e592700 (LWP 5993) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
4 Thread 0x7f789dd91700 (LWP 5994) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
5 Thread 0x7f7899590700 (LWP 5995) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
6 Thread 0x7f7896d8f700 (LWP 5996) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
7 Thread 0x7f789458e700 (LWP 5997) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
8 Thread 0x7f7891d8d700 (LWP 5998) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
9 Thread 0x7f788f58c700 (LWP 5999) "python" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
...more
when I tried to look into the thread, it seems my gdb is not well set:
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fe775e0a700 (LWP 6052))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
The comparison I run with the native docker, it just display:
(gdb) info threads
No threads.
It proves that threads were not properly collected on k8s.
Steps to reproduce
import time
import cv2
import numpy as np
if __name__ == "__main__":
with open("test.jpeg", "rb") as f:
image = f.read()
image_arr = np.fromstring(image, np.uint8)
image_dec = cv2.imdecode(image_arr, cv2.IMREAD_COLOR)
print(f"Image shape is {image_dec.shape}")
while True:
print("Sleep start")
time.sleep(3)
print("Sleep end")