Describe the bug

When the desktop viewer window is minimised or on some machine bring to background of other windows, there are data seemingly being cached by the rerun process.

When the viewer window is bring to foreground again, i can see it being fast forward in replay, before it goes back to realtime again.

If i leave the window minimised for too long, eventually it dries up the memory on system and hangs the system

To Reproduce
Steps to reproduce the behavior:

1. Run with rerun --drop-at-latency 500ms --server-memory-limit 200MB --memory-limit 200MB
2. Stream image data over time
3. Minimise the viewer window
4. Observer using htop to prove that memory increases steadily

Expected behavior

Memory usage should be bounded by --server-memory-limit 200MB --memory-limit 200MB

Screenshots

Backtrace

Desktop (please complete the following information):

• OS: Linux 20.04 / 22.04

Rerun version

0.15.1 or 0.18.0 tested

Additional context