This is a nice idea discussed in #15248. I think it's better to discourage build isolation with extra-build-dependencies, but this will "do the right thing" in some common cases, like flash-attn.