Skip to content

Timing with resources#35

Merged
TomAugspurger merged 3 commits intodask:masterfrom
leej3:timing_with_resources
Feb 27, 2020
Merged

Timing with resources#35
TomAugspurger merged 3 commits intodask:masterfrom
leej3:timing_with_resources

Conversation

@leej3
Copy link
Copy Markdown

@leej3 leej3 commented Feb 27, 2020 •
edited by TomAugspurger
Loading

@TomAugspurger, this will work once @lr4d has merged his changes to allow configurable work stealing intervals for the scheduler (dask/distributed#3523).

It looks like on the branch to fix job stealing with arbitrary resources (dask/distributed#3069) the scheduler is slower once resources are specified but is still much better than master (or what will soon be master... add_work_stealing_callback_config from lr4d).

On add_work_stealing_callback_config:

========== ========= ========= ========= =========
--                      steal_interval
---------- ---------------------------------------
resource     0.01      0.1        1        100
========== ========= ========= ========= =========
  1        1.08±0s   1.06±0s   1.07±0s   1.07±0s
 None      436±0ms   430±0ms   436±0ms   436±0ms
========== ========= ========= ========= =========

On dask/distributed#3069:

========== ========= ========= ========= =========
--                      steal_interval
---------- ---------------------------------------
resource     0.01      0.1        1        100
========== ========= ========= ========= =========
  1        625±0ms   542±0ms   547±0ms   644±0ms
 None      428±0ms   425±0ms   431±0ms   438±0ms
========== ========= ========= ========= =========

leej3 and others added 3 commits February 26, 2020 15:51
test stealing time with and without resources specified
7386ec1
parameterize worker timing benchmark
9b85703 
paramaterize across stealing interval and
resource specification
make sure that the params are appropriately modifying the scheduler
@TomAugspurger
trigger ci
e689ff2
@TomAugspurger
Copy link
Copy Markdown
Member

TomAugspurger commented Feb 27, 2020

Thanks! That's @lr4d's PR is in. I pushed an empty commit to see if CI passes.

@TomAugspurger
Copy link
Copy Markdown
Member

TomAugspurger commented Feb 27, 2020

slower once resources are specified but is still much better than master

And just to be clear, the comparision we really care about is this dask/distributed#3069 vs. master, rather than with resources vs. without resources. Looking at your numbers it seems like we're faster (or at least not slower) in both cases (with resource restrictions and without)

@TomAugspurger TomAugspurger merged commit 8e22c6a into dask:master Feb 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@leej3 @TomAugspurger