Thanks, I'll have a look next week but people who are more familiar with codecov are welcome to review too!

Same here!

Just to check, what is the motivation for this? Ideally we would measure coverage on all jobs but we don't for performance reasons as it adds a significant overhead. I'm curious what we gain by adding it to this job?

There was confusion when a test that only runs in a minimal dependency setting gets reported as not covered even though there is a test.

#12192 (comment)

Ok thanks for the background! Maybe a comment should be added in the CI config to explain this.

Is the addition of -clocale needed? If not, maybe just add -cov?

I think the clocale can be removed, at least #12192 (comment) does not require it.

Maybe a comment should be added in the CI config to explain this.

Done. Added a short note about running tests without scipy.

Does not look like the overhead is all that large – only a few minutes difference in testing times, at most.
I am a bit puzzled by the py38-test-alldeps job BTW, which is running with --open-args, but contrary to the comments does not seem to take any longer to run either.
By comparison, running astropy.test(open_files=True) locally takes over 1 hour compared to ~10 min without – or is that just another macOS thingy?

Wow, an hour! I am not sure... I don't think I ever tried to run open_files=True with the astropy test runner. I can try it on other OSes when I get the chance.

Yup, I see 1.5 hours on Linux. Must be the overhead of checking for dangling file pointers in the test runner. I don't see it being stuck on any one job. Not sure what we can do about that.

17767 passed, 201 skipped, 66 xfailed in 5241.37s (1:27:21)