It should be possible with pytest_xdist_auto_num_workers to:

• Read numbers of cores
• Read total system memory
• Determine if CPython or PyPy

And come up with a reasonable number of cores.

Originally posted by @SamWilsn in #2120 (comment)

pytest docs: CPU parallelization

Right now pytest uses the -n auto flag that directs pytest-xdist to use as many processes as the computer running the tests has physical CPU cores, up to the limit set by --maxprocesses. The maximum number of processes is currently set to specific values based on our experience of the CI machine's performance.

We could likely increase throughput by switching to the -n logical flag which would use the max number of logical CPU cores, however, before we do that we need to dynamically adjust the --maxprocesses cap based on the system that is running the tests. To do that, we will have to figure out how to determine the available memory on the system and properly adjust to ensure we don't run out.

We should also investigate using the --dist worksteal flag for impact on test times which could handle re-assigning jobs to workers who have completed their test queue more efficiently.