#4941 (on features) changes testdir to set $HOME to the tempdir in Tempdir.__init__(), but I think it should only be done in the runpytest methods - otherwise you will not get your pdb config in any test that uses testdir.
OTOH this was the main reason - to make tests not output something unexpected etc, and it would still mean that the config is missing from inner set_trace.

Not sure what to do about this, but the behaviour should be probably different from set_trace being used in tests (test_pdb.py mostly), and in other cases.
Maybe pytest's tests should use a custom set_trace, which would handle $HOME - but just doing it in the runpytest methods might be a good first step / improvement.