I believe there is no need to actually have a custom threadpool and the ability to "secede" from the threadpool, i.e. to adjust the allowed thread count while running.

I believe this functionality causes confusion for users and it doesn't work as intended.

Right now we do have two mechanisms in place to control the number of running Tasks. There is the Threadpool limit which is, by default, set to Worker.state.nthread and there is our state machine that controls how many tasks are allowed to be in the state executing, i.e. how many tasks were already submitted to the executor without it being returned.

Using worker.secede performs two actions

1. Secede from the threadpool, i.e. increase the number of allowed threads in the threadpool by one. In fact, it removes the current thread from the threadpool tracking and likely causes it to leak.
2. Transition the task to state long-running which tells the worker state that this currently executing task should not count towards the limit nthreads

Only one of these mechanisms is required to fulfill our requirements for

• Limit concurrently running tasks
• Allow targeted, manual override of running tasks by seceding the current task

If we look further into how this mechanism can be used incorrectly, we will notice that there is no worker.rejoin method (see #5882) and there is no actual way to re-enter the threadpool (in a safe way) and update the worker state at the same time. Users may resort to using the threadpoolexecutor.rejoin which implements a reentering of the thread in the threadpool but does not implement a reentrance to the worker state by transitioning the task back to executing. This causes the threadpool to be already saturated even though the number of tasks in state executing that are actually running on the threadpool is smaller than Worker.nthreads, i.e. Worker.execute will actually block without submitting any tasks to the executor.

This error is actually surprisingly easy to do since we're importing worker.secede but threadpool.rejoin in our top-level __init__.py

A similar problem occurs if users are using [threadpool.secede] instead of worker.secede. While this actually frees up a slot on the threadpool, the worker would never transition any more tasks and we wouldn't even increase concurrency, i.e. this would not have any effect. This import is not exposed top-level.

The bottom line is that we should definitely not expose threadpool.secede or threadpool.rejoin to the public API since the usage is most likely wrong unless people are using our threadpool without using distributed and know exactly what they are doing.

At the same time, updating the threadpool to a more recent version has been shown to be difficult (#5893)

If nothing surprising pops up on CI I would suggest to drop this entirely