After messing around with it for a while, I solved this problem by realising I could launch the debugger via --user:"path/to/file" to just override the defaults.

For a UX flow improvement it might be nice if the debugger left back off on the last known user & project? Then the user doesn't have to do this step.