Add set of tasks that should not be added to stack in order#3303
Merged
mrocklin merged 1 commit intodask:masterfrom Mar 21, 2018
Merged
Add set of tasks that should not be added to stack in order#3303mrocklin merged 1 commit intodask:masterfrom
mrocklin merged 1 commit intodask:masterfrom
Conversation
Contributor
|
That looks like a pretty simple fix! I will try to find time to test this branch on my real-world problem within the next few days |
Member
Author
|
Merging this tomorrow if there are no further comments |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously we would re-add a task to the stack many times if it had many dependencies.
We now maintain a set of tasks that should not be re-added and check it.
This results in a significant reduction of costs in order in cases where a single output has
many input dependencies.
Real-world cause is here: pangeo-data/pangeo#150 (comment)
I don't know of a nice way to test this. This is one of those situations where having benchmarks directly within the repository would be convenient.
flake8 daskdocs/source/changelog.rstfor all changesand one of the
docs/source/*-api.rstfiles for new API