Added resource stealing benchmark#22
Merged
TomAugspurger merged 6 commits intodask:masterfrom Nov 7, 2019
Merged
Conversation
benchmarks the improvement on dask/distributed#3069
mrocklin
reviewed
Nov 7, 2019
|
|
||
| def setup(self): | ||
| cluster = LocalCluster(n_workers=1, threads_per_worker=1, | ||
| resources={"resource": 1}, worker_class=Worker) |
Member
There was a problem hiding this comment.
For speed reasons we might want to use processes=False in cases like this. It also helps considerably with cleanup.
mrocklin
reviewed
Nov 7, 2019
| cluster = LocalCluster(n_workers=1, threads_per_worker=1, | ||
| resources={"resource": 1}, worker_class=Worker) | ||
| spec = copy.deepcopy(cluster.new_worker_spec()) | ||
| del spec[1]['options']['resources'] |
Member
There was a problem hiding this comment.
I encourage folks to take a look at using SpecCluster directly. It's not terrible to use manually, and make this sort of thing more explicit.
mrocklin
reviewed
Nov 7, 2019
| workers = list(info['workers']) | ||
| futures = client.map(slowinc, range(10), | ||
| delay=0.1, resources={"resource": 1}) | ||
| client.cluster.scale(len(workers) + 1) |
Member
There was a problem hiding this comment.
The 100ms cost here is likely to dwarf the task overhead that we're looking to find here, which is likely to be in the 10us range.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
benchmarks the improvement on dask/distributed#3069.
Will verify that we have a benchmark for the common case of no restrictions.