Skip to content

Conversation

@Carreau
Copy link
Member

@Carreau Carreau commented Feb 14, 2025

Right now the cleanup was mostly done at process exit time,
but this leads to issues in downstream test that start to have dozen
of threads blocking, and make finding deadlocks harder

This PR try to cleanup the cleaning logic.

  • Try to seprate saving thread from history manager
  • Make sure to not keep reference to history manager in multiple places.
  • att a utility to refuse to create more then N
    concurrent history manager HistorySavingThread
    instances.
  • default to 1 in conftest.py
  • allow to set it to N via pytest fixture.

Some of this should likely be either context managers, or finalisers.

@Carreau Carreau force-pushed the hmclean branch 5 times, most recently from ed26da6 to 49297b3 Compare February 15, 2025 09:03
@Carreau Carreau changed the title Hmclean HistoryManager cleanup. Feb 15, 2025
Right now the cleanup was mostly done at process exit time,
but this leads to issues in downstream test that start to have dozen
of threads blocking, and make finding deadlocks harder

This PR try to cleanup the cleaning logic.

 - Try to seprate saving thread from history manager
 - Make sure to not keep reference to history manager in multiple places.
 - att a utility to refuse to create more then N
   concurrent history manager HistorySavingThread
   instances.
   - default to 1 in conftest.py
   - allow to set it to N via pytest fixture.

Some of this should likely be either context managers, or finalisers.
@Carreau Carreau added this to the 9.0 milestone Feb 15, 2025
@Carreau Carreau merged commit 14482f8 into ipython:main Feb 15, 2025
14 of 18 checks passed
@davidbrochart
Copy link

@Carreau You mentioned here that this PR cleans the history manager threads at garbage collection.
Would it be possible to have a more explicit and deterministic way of doing so?

@Carreau
Copy link
Member Author

Carreau commented Feb 25, 2025

Not without a major refactor and breaking a lot of APIs. Basically you would need InteractiveShell itself to be a contextmanager.

There are a number of other things that should be context managers. Some objects are doing crazy things like having reset() methods instead of having consumer just delete the current and create a new one – which made sens as a resource consumption 10 years ago, but not now anymore.

In conftest.py you can set HistorySavingThread's class attribute _max_inst to an integer and it will prevent creating more that that number of instances, so you can track the leaking ones.

See how I'm doing it in this PR, and the fixture I use to temporarily increase the number of HistoryManager for specific tests.

@davidbrochart
Copy link

davidbrochart commented Feb 25, 2025

I think doing cleanup at garbage collection is very brittle because the object can only be garbage collected if there is no reference to it anymore, which is out of control.
I am not necessarily talking about using context managers, but at least providing a method that stops the thread, which would not be a breaking change.

@Carreau
Copy link
Member Author

Carreau commented Feb 25, 2025

There are already methods that stop the thread, you can already call history_manager.save_thread.stop(). Really there should be finalizers that should:

  • be explicitly called in a context manager at best
  • otherwise in a .destroy() or similar method
  • if not, called at gc, but emit a ResourceWarning
  • if not called at atexit (and also emit a resource warning)

finalizer will make sure those get called at most once, or be no-op, but still requires a lot of refactor and tracking.

Like right now, closing the thread while there are still live instance of history manager meas that we likely have a bug. So yes I agree with you, but it will need some manpower to make sure the code is sound (right now it definitely isn't), and detecting leaks will help us making it sound.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants