[core] Spill at least the object fusion size instead of at most#22750
Merged
stephanie-wang merged 23 commits intoray-project:masterfrom Apr 5, 2022
Merged
[core] Spill at least the object fusion size instead of at most#22750stephanie-wang merged 23 commits intoray-project:masterfrom
stephanie-wang merged 23 commits intoray-project:masterfrom
Conversation
rkooo567
approved these changes
Mar 2, 2022
Contributor
rkooo567
left a comment
There was a problem hiding this comment.
Didn't review the PR, but I assume the bug was fixed..
80758d4 to
8507965
Compare
Member
|
Contributor
Author
|
@rkooo567 I updated this PR a bit to address another issue I found where we're spilling at most the fusion size instead of at least. Can you take a look at the C++ changes and re-approve? |
rkooo567
requested changes
Apr 1, 2022
Contributor
rkooo567
left a comment
There was a problem hiding this comment.
Temporarily request changed when I review the new change (will do this sun)
rkooo567
approved these changes
Apr 4, 2022
Contributor
rkooo567
left a comment
There was a problem hiding this comment.
I think we should update the docstring, but other than that, it lgtm
| int64_t counts = 0; | ||
| while (bytes_to_spill <= num_bytes_to_spill && it != pinned_objects_.end() && | ||
| counts < max_fused_object_count_) { | ||
| while (it != pinned_objects_.end() && counts < max_fused_object_count_) { |
Contributor
There was a problem hiding this comment.
Can you update the docstring of SpillObjectsOfSize to explain semantic changes?
edoakes
pushed a commit
to edoakes/ray
that referenced
this pull request
Apr 7, 2022
…project#22750) Copied from ray-project#22571: Whenever we spill, we try to spill all spillable objects. We also try to fuse small objects together to reduce total IOPS. If there aren't enough objects in the object store to meet the fusion threshold, we spill the objects anyway to avoid liveness issues. However, currently we spill at most the object fusion size when instead we should be spilling at least the fusion size. Then we use the max number of fused objects as a cap. This PR fixes the fusion behavior so that we always spill at minimum the fusion size. If we reach the end of the spillable objects, and we are under the fusion threshold, we'll only spill it if we don't have other spills pending too. This gives the pending spills time to finish, and then we can re-evaluate whether it's necessary to spill the remaining objects. Liveness is also preserved. Increases some test timeouts to allow tests to pass.
stephanie-wang
added a commit
to stephanie-wang/ray
that referenced
this pull request
May 11, 2022
…st (ray-project#22750)" This reverts commit 1c972d5.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Copied from #22571:
Whenever we spill, we try to spill all spillable objects. We also try to fuse small objects together to reduce total IOPS. If there aren't enough objects in the object store to meet the fusion threshold, we spill the objects anyway to avoid liveness issues. However, currently we spill at most the object fusion size when instead we should be spilling at least the fusion size. Then we use the max number of fused objects as a cap.
This PR fixes the fusion behavior so that we always spill at minimum the fusion size. If we reach the end of the spillable objects, and we are under the fusion threshold, we'll only spill it if we don't have other spills pending too. This gives the pending spills time to finish, and then we can re-evaluate whether it's necessary to spill the remaining objects. Liveness is also preserved.
Increases some test timeouts to allow tests to pass.