Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.25%. Comparing base (905b0ca) to head (1a748e1).
Report is 3 commits behind head on main.

@@            Coverage Diff             @@
##             main    #1589      +/-   ##
==========================================
- Coverage   95.27%   95.25%   -0.03%     
==========================================
  Files          45       45              
  Lines        7708     7715       +7     
==========================================
+ Hits         7344     7349       +5     
- Misses        364      366       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

I was just looking at the multiprocessing fork method deprecation warning, since they also show up in SciPy tests. Given that the default method will change in Python 3.14 (https://docs.python.org/3.13/whatsnew/3.13.html#pending-removal-in-python-3-14), did you consider always specifying the start method, and/or defaulting to forkserver? It's actually pretty unclear to me (even after reading python/cpython#84559 (comment) and other comments in that issue) whether we're supposed to fix it in libraries or let users deal with it themselves by setting multiprocessing.set_start_method in case they don't want the defaults to switch from under them in 3.14

Thanks for the pointers, so indeed the change is that on Linux in CPython 3.14, the default multiprocessing start method will change from 'fork' to a better default (I haven't read enough to figure out what the new default is 😅)

To quote the relevant section for more precision:

multiprocessing: The default start method will change to a safer one on Linux, BSDs, and other non-macOS POSIX platforms where 'fork' is currently the default (gh-84559). Adding a runtime warning about this was deemed too disruptive as the majority of code is not expected to care. Use the get_context() or set_start_method() APIs to explicitly specify when your code requires 'fork'. See Contexts and start methods.

In joblib 'multiprocessing' backend case, I am not sure either what we should do. I see broadly three directions:

1. use multiprocessing with fork i.e. the default until Python 3.13, at least code that was working with joblib before will still work the same, for example interactive functions (i.e. defined in __main__).
2. let the warning through and add some documentation to tell users to set it explicitly the context depending on their use case.
3. swallow the warning because we are expecting most users are happy with the default start method even if it changes in Python 3.14. Document the caveat when Python 3.14 is released to say that if you had code that was working before, here is what you need to keep it working.

I would lean slightly towards 3. maybe? Probably not 1. because my understanding is that joblib 'multiprocessing' backend should respect the start method if set by the user.

To be honest I am not that sure how much the joblib 'multiprocessing' backend is used ... but maybe a significant portion of these users do it because they rely on the fork behaviour, hard to tell ...

cc @tomMoral and @ogrisel for informed opinions on this topic

I haven't read enough to figure out what the new default is

My understanding is: very likely 'forkserver', but the devs reserved the right to change that decision during the 3.14 alpha phase.

Doing a Parallel with loky followed by a Parallel by multiprocessing yields a DeprecationWarning because I guess loky creates thread and doing a fork of a multi-threaded warning may lead to issues. Maybe we should swallow warning in the test? Not sure if there are use cases where using both loky and multiprocessing and what to do in this case.

I agree we might want the swallow this warning in our test. This is quite a synthetic case that should not happen much in practice, so we can ignore and leave the deprecation warning go through in joblib.

did you consider always specifying the start method, and/or defaulting to forkserver? It's actually pretty unclear to me (even after reading python/cpython#84559 (comment) and other comments in that issue) whether we're supposed to fix it in libraries or let users deal with it themselves by setting multiprocessing.set_start_method in case they don't want the defaults to switch from under them in 3.14

I think I would let the multiprocessing users chose their own start method. I would not change the default because:

• it would be a behavioral breaking change for people who rely on the semantics of the fork start method;
• joblib's default loky backend is "better" than multiprocessing + forkserver because it allows to call functions defined interactively in the __main__ module or as local variable under the scope of another function. So I would not push users towards a solution that does not allow that.

Once CPython 3.14 has settled down on its plan, we can adapt joblib accordingly to adapt the tests for CPython 3.14 and later but I would not change joblib's default behavior for earlier Python versions.

3. swallow the warning because we are expecting most users are happy with the default start method even if it changes in Python 3.14. Document the caveat when Python 3.14 is released to say that if you had code that was working before, here is what you need to keep it working.

+1. I just assume that "swallow the warning" is only done in when running the tests, not in general in joblib.

OK, CI is green and this is ready for another review round!

I have mostly:

• added a free-threaded Python 3.13 is in Azure rather than Github action (Github action was easier originally to get started since they have a deadsnake/action but this was always the goal to go back to Azure eventually)
• ignored the fork of multi-threaded process warning in the test

Thanks @lesteve !

This is surprisingly a small diff 😅

I guess the issue are more likely to happen when using free-threaded + joblib multithreading backend + some other library.

Next thing I want to do is run the joblib test suite with default backend switched to threading. I think locally I have seen some errors but I need to investigate more.

After this trying to run the scikit-learn test suite with default backend switched to threading.