Fix the resource key representation before sending graphs (#2716)#2733
Merged
mrocklin merged 5 commits intodask:masterfrom Jun 3, 2019
Merged
Fix the resource key representation before sending graphs (#2716)#2733mrocklin merged 5 commits intodask:masterfrom
mrocklin merged 5 commits intodask:masterfrom
Conversation
Convert resource key toples to a string representation before they are submitted to the scheduler. The commit is intended to fix dask#2716.
Member
|
Thanks for the PR @Spiegel0 ! It would be good to include a test in this PR that would have failed before this fix. There is some information here: http://distributed.dask.org/en/latest/develop.html#writing-tests Perhaps something like the following, with the example that you shared in the original issue filled in. @gen_cluster(client=True)
def test_resources_str(c, s, a, b):
a.set_resources({"MyRes": 1000})
df = ...
df.persist(resouces=...)
yield wait(df)
ts = s.tasks[tokey(df.__dask_keys__()[0])]
assert ts.resources == |
Contributor
Author
|
You are right, a test case would be very beneficial. Thank you for providing me the code snippet! I currently don't have access to my development environment but I'll try to implement the test case on Monday or Tuesday. (I guess its a good opportunity to get started with the Dask distributed internals.) I'll let you know as soon as I have the test case ready. |
The test case persists the result of a tiny DataFrame operation and checks the resource restrictions.
Convert resource key toples to a string representation before they are submitted to the scheduler. The commit is intended to fix dask#2716.
…d into issue-2716-resources
Member
|
Would you have another chance to take a look, @mrocklin? Looks like there is now a test case. Travis CI is passing (with the exception of Python 2.7, which we may not care about here). AppVeyor is failing, but it is unclear (at least to me) if the failures are related. |
Member
|
Looks good. Merging. Thanks @Spiegel0 ! |
Contributor
Author
|
Thank you very much for the assistance and for merging the PR @mrocklin, @jakirkham and @TomAugspurger! |
muammar
added a commit
to muammar/distributed
that referenced
this pull request
Jun 12, 2019
* upstream/master: (58 commits) Add unknown pytest markers (dask#2764) Delay lookup of allowed failures. (dask#2761) Change address -> worker in ColumnDataSource for nbytes plot (dask#2755) Remove module state in Prometheus Handlers (dask#2760) Add stress test for UCX (dask#2759) Add nanny logs (dask#2744) Move some of the adaptive logic into the scheduler (dask#2735) Add SpecCluster.new_worker_spec method (dask#2751) Worker dashboard fixes (dask#2747) Add async context managers to scheduler/worker classes (dask#2745) Fix the resource key representation before sending graphs (dask#2716) (dask#2733) Allow user to configure whether workers are daemon. (dask#2739) Pin pytest >=4 with pip in appveyor and python 3.5 (dask#2737) Add Experimental UCX Comm (dask#2591) Close nannies gracefully (dask#2731) add kwargs to progressbars (dask#2638) Add back LocalCluster.__repr__. (dask#2732) Move bokeh module to dashboard (dask#2724) Close clusters at exit (dask#2730) Add SchedulerPlugin TaskState example (dask#2622) ...
muammar
added a commit
to muammar/distributed
that referenced
this pull request
Jul 18, 2019
* upstream/master: (43 commits) Add unknown pytest markers (dask#2764) Delay lookup of allowed failures. (dask#2761) Change address -> worker in ColumnDataSource for nbytes plot (dask#2755) Remove module state in Prometheus Handlers (dask#2760) Add stress test for UCX (dask#2759) Add nanny logs (dask#2744) Move some of the adaptive logic into the scheduler (dask#2735) Add SpecCluster.new_worker_spec method (dask#2751) Worker dashboard fixes (dask#2747) Add async context managers to scheduler/worker classes (dask#2745) Fix the resource key representation before sending graphs (dask#2716) (dask#2733) Allow user to configure whether workers are daemon. (dask#2739) Pin pytest >=4 with pip in appveyor and python 3.5 (dask#2737) Add Experimental UCX Comm (dask#2591) Close nannies gracefully (dask#2731) add kwargs to progressbars (dask#2638) Add back LocalCluster.__repr__. (dask#2732) Move bokeh module to dashboard (dask#2724) Close clusters at exit (dask#2730) Add SchedulerPlugin TaskState example (dask#2622) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convert resource key tuples to a string representation before they are submitted to the scheduler.
The commit is intended demonstrate a quick-fix of #2716. I'm not sure whether it breaks something else because I wasn't able to successfully run all test-cases, yet.