Skip to content

Implement work-stealing scheduler#862

Merged
nicoddemus merged 3 commits intopytest-dev:masterfrom
amezin:workstealing
Jan 11, 2023
Merged

Implement work-stealing scheduler#862
nicoddemus merged 3 commits intopytest-dev:masterfrom
amezin:workstealing

Conversation

@amezin
Copy link
Collaborator

@amezin amezin commented Jan 5, 2023

Closes #858

@nicoddemus
Copy link
Member

nicoddemus commented Jan 6, 2023

Thanks @amezin!

I will review it in the next few days.

You have made several contributions already, would you like to help maintain the project and become a contributor?

@amezin
Copy link
Collaborator Author

amezin commented Jan 6, 2023

What does "becoming a contributor" mean? GitHub already shows the "Contributor" badge...

@nicoddemus
Copy link
Member

nicoddemus commented Jan 7, 2023

Basically write access, being able to review/merge PRs, and making releases.

nicoddemus
nicoddemus reviewed Jan 7, 2023
View reviewed changes
Copy link
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good!

Left a few minor comments which would be nice to get in as well.

Did you test this new mode in a large test suite?

@nicoddemus
Copy link
Member

nicoddemus commented Jan 7, 2023

Feel free to use type-annotations as well, as we run mypy on the repository. 👍

amezin added 3 commits January 8, 2023 05:39
@amezin
WorkerInteractor: handle commands immediately
b6d6c4d 
This will be necessary later for 'steal' command.

Luckily, all tests still pass after this change.
@amezin
WorkerInteractor: implement "steal" command
26d94ce
@amezin
Implement work-stealing scheduling
df71225 
Closes pytest-dev#858
@amezin
Copy link
Collaborator Author

amezin commented Jan 8, 2023

Did you test this new mode in a large test suite?

https://github.com/ddterm/gnome-shell-extension-ddterm/actions/runs/3843994508/jobs/6546739410

Speed/time seems to be indistinguishable from load - either when the suite is split to multiple jobs, or when I run all tests at once (~4k tests total). On GitHub CI, run to run variance is higher than load vs worksteal difference. When I run the same suite locally, I get maybe 5% speedup compared to load, but nothing really noticeable.

@amezin
Copy link
Collaborator Author

amezin commented Jan 8, 2023

Basically write access, being able to review/merge PRs, and making releases.

Yes, I'm interested. But can't promise I'll be consistently active.

@nicoddemus
Copy link
Member

nicoddemus commented Jan 9, 2023

Yes, I'm interested. But can't promise I'll be consistently active.

Sure, no problem, we have no such expectations. 👍

nicoddemus
nicoddemus approved these changes Jan 9, 2023
View reviewed changes
Copy link
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amezin!

Will merge this in the next few days to give the chance for others to review too. 👍

@nicoddemus nicoddemus merged commit d1dfad3 into pytest-dev:master Jan 11, 2023
@amezin amezin mentioned this pull request Jan 11, 2023
Closed
@cpnat cpnat mentioned this pull request Feb 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicoddemus nicoddemus nicoddemus approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement work-stealing scheduler

2 participants

@amezin @nicoddemus