my testing strategy is not ideal in that it repeats fixture code but I couldn't find a smart way to avoid that. In particular, parametrized fixtures didn't seem applicable here given the subtle differences between each setup.
These tests are also very likely thread unsafe as they all modify a global state (os.environ). This isn't a problem right now but could be in the future when we start experimenting with free-threaded Python.

I am not sure about using set_temp_config here. It was meant for temporary stuff, not suitable for cases like you want to permanently change the path for a machine.

Currently your patch only deals with cache, but config also should be consistent:

HOME
  |__ .astropy
        |___ cache
        |___ config

I think the ask was to change where HOME points to, which affects both cache and config.

What I had in mind, I think, was simpler, we add one more if in _get_dir_path to check for the new env var first before going to XDG env var (for both cache and config).

Additionally, the user doc would need to make clear of this new option and how it might catch users unawares if they accidentally have those new env var defined, etc.

Also have to make sure we don't break https://docs.astropy.org/en/latest/config/index.html#customizing-config-location-in-affiliated-packages in the process.

This bit is from #17554. I'll move your comment and answer there !
to clarify, only the second commit on this PR is to be reviewed here.

Actually never mind, I think I should answer here because the second half of your comment makes the most sense in this context;

I am not sure about using set_temp_config here.

Note that it is already being used. I'm just moving the function around to make it a class method instead of a floating function that takes a class as one of its arguments.

Currently your patch only deals with cache, but config also should be consistent

You're right, it seems I was half-remembering a conversation from #17507. I'll add ASTROPY_CONFIG_DIR too once I stabilized CI (listed both tasks as todos on the original post).

What I had in mind, I think, was simpler, we add one more if in _get_dir_path to check for the new env var first before going to XDG env var (for both cache and config).

that is actually exactly what I did in this branch. It's easier to see if you just look at the second commit.

It would be simplest both for us and for the users if the effect of the new Astropy-specific environment variables would be as similar as possible to the effect of the already available XDG environment variables. My (naive) expectation is that the code that handles that should look something like

for env_dir in (cls._astropy_env_dir, cls._xdg_env_dir):
  ...

Why is there a separate code block for the new functionality?

I can't remember what my reasoning was, maybe I was just trying to get the most simple behavior right from the get go; the existing logic for XDG variables is quite complex and I probably thought I couldn't test it all efficiently.

cls._temp_path being handled after the Astropy environment variables means that set_temp_cache and set_temp_config will fail silently at runtime if the corresponding Astropy environment variable is used. The fact that this is not being caught be the tests means that we need to add tests that would check which directory customization mechanism has priority if multiple are used simultaneously.

good point

Why are you implementing this functionality yourself instead of making use of pytest monkeypatching?

I try to reserve mock/monkeypatching as last resorts because I think they are not well understood and tend to effectively make tests harder to read or refactor (see for instance #16659).

You have added several new tests, but you have not modified any of the existing ones. I would have thought that the tests involving the XDG environment variables could be rewritten to be parametrized tests that check both the XDG and the new Astropy environment variables.