Skip to content
This repository was archived by the owner on Mar 6, 2026. It is now read-only.

test: adds pytest-xdist to speed up processing of CI/CD checks#2153

Merged
chalmerlowe merged 8 commits intomainfrom
test-xdist-experimentation
Apr 11, 2025
Merged

test: adds pytest-xdist to speed up processing of CI/CD checks#2153
chalmerlowe merged 8 commits intomainfrom
test-xdist-experimentation

Conversation

@chalmerlowe
Copy link
Copy Markdown
Collaborator

DO NOT MERGE.
WIP experiments with pytest-xdist.

@chalmerlowe chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 2, 2025
@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Apr 2, 2025
@product-auto-label product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels Apr 3, 2025
@chalmerlowe chalmerlowe removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 11, 2025
@chalmerlowe chalmerlowe changed the title experimentation using pytest-xdist test: adds pytest-xdist to speed up processing of CI/CD checks Apr 11, 2025
@chalmerlowe
Copy link
Copy Markdown
Collaborator Author

@Linchin

Using pytest-xdist for tests that run under pytest shortens the time for a number of our CI/CD checks. In this image, darker green means more time savings. For some checks we cut total time by five minutes.

Our two worst offenders do not benefit, because:

  • PRESUBMIT: has nox sessions internally and though each is less than 2 minutes long, each goes through a virtualenv setup, package install, etc and does so sequentially thus by the time we finish ~10 sessions, we have aggregated 13 minutes). This should be broken into individual CI/CD checks (out of scope).
  • LINTING-TYPING: does not use pytest, instead these tests run using mypy, pytype, etc. pytype is the long pole in tent at 10 minutes. Despite this, these should be broken into individual CI/CD checks to allow more of them to run in parallel (out of scope). It may be possible to provide a wrapper to run pytype against multiple files in parallel using multiprocessing or subprocess modules (out of scope).
Screenshot 2025-04-11 at 6 37 57 AM

@chalmerlowe chalmerlowe marked this pull request as ready for review April 11, 2025 11:47
@chalmerlowe chalmerlowe requested a review from a team April 11, 2025 11:47
@chalmerlowe chalmerlowe requested review from a team as code owners April 11, 2025 11:47
@chalmerlowe chalmerlowe requested review from nicain and tswast April 11, 2025 11:47
# Run py.test against the unit tests.
session.run(
"py.test",
"-n=auto",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the default is the number of CPUs on the instance, right? I recall in bigframes we set this a bit higher than that and it seemed to help.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you are correct, that the default is the number of CPUs on the instance.

Copy link
Copy Markdown
Contributor

@Linchin Linchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the tests a lot faster!

@chalmerlowe chalmerlowe merged commit 1cabacb into main Apr 11, 2025
17 checks passed
@chalmerlowe chalmerlowe deleted the test-xdist-experimentation branch April 11, 2025 19:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: bigquery Issues related to the googleapis/python-bigquery API. size: s Pull request size is small.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants