I just remembered that loky does cache the number of physical cores. And at the time I even wrote a comment saying that it should not change during the lifetime of a program. Unfortunately it doesn't cache the total number of cores.
So I'm in favor of extending the cache for both cpu counts in loky, and to add the cache mechanism in scikit-learn at the same time.

@jeremiedbb I updated the PR.