[Core] Fix placement group leaks#42942
Merged
rkooo567 merged 5 commits intoray-project:masterfrom Feb 8, 2024
Merged
Conversation
Contributor
|
can we have a unit test, e.g. start a pg and kill it right away, and see if there's any leak? |
added 2 commits
February 2, 2024 22:29
Contributor
Author
|
@rynewang I am figuring that out. This requires very subtle timing (indeed, I could repro when I created so many pgs only), so writing an unit test is not trivial. Writing cpp unit test is even harder because of how our code is structured. |
rkooo567
commented
Feb 6, 2024
| // Retry 10 * worker registeration timeout to avoid race condition. | ||
| // See https://github.com/ray-project/ray/pull/42942 | ||
| // for more details. | ||
| /*max_retry*/ RayConfig::instance().worker_register_timeout_seconds() * 10, |
Contributor
Author
There was a problem hiding this comment.
Q: should we do exponential backoff? Right now, we retry every 1 second.
Contributor
Author
|
@jjyao it is ready to be reviewed |
Contributor
Author
|
Q: Right now ,we retry for 10 minutes with 1 second interval. Should we
|
Contributor
When can it happen? |
Contributor
Author
When a worker is not started yet! |
jjyao
reviewed
Feb 7, 2024
| const rpc::CancelResourceReserveReply &reply) { | ||
| RAY_LOG(DEBUG) << "Finished cancelling the resource reserved for bundle: " | ||
| << bundle_spec->DebugString() << " at node " << node_id; | ||
| [this, bundle_spec, node_id, node, max_retry, current_retry_cnt]( |
Contributor
There was a problem hiding this comment.
Where do you use the max retry
Contributor
Author
There was a problem hiding this comment.
oops.. let me fix this. I will also add an unit test
|
|
||
| void ClusterResourceManager::AddOrUpdateNode(scheduling::NodeID node_id, | ||
| const NodeResources &node_resources) { | ||
| RAY_LOG(DEBUG) << "Update node info, node_id: " << node_id.ToInt() |
Contributor
Author
There was a problem hiding this comment.
This seems useless and too verbose. I can bring it back if you think it is necessary
Contributor
Author
|
@jjyao it'd be great if you can approve the PR. I will add changes and merge it |
jjyao
approved these changes
Feb 7, 2024
ratnopam
pushed a commit
to ratnopam/ray
that referenced
this pull request
Feb 11, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well. Signed-off-by: Ratnopam Chakrabarti <ratnopamc@yahoo.com>
rkooo567
added a commit
to rkooo567/ray
that referenced
this pull request
Feb 12, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well.
aslonnie
pushed a commit
that referenced
this pull request
Feb 13, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started".
It fixes the issue by retrying cancellation. This also means
Alternatively, to improve the consistency, we can also do
I chose the current solution as it is needed for a network issue as well.
Related issue number
Closes #26761
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.