I think we should make externals private to make it easier to stop vendoring things. I don't think users should use the joblib that comes with sklearn.
I don't think externals should have ever been public, but in particular in light of #11166 and #11115 I think now is a good time to deprecate the public version. Otherwise we might end up having to provide backports for 2.7 when we don't support it any more...