-
Notifications
You must be signed in to change notification settings - Fork 7.4k
[Core] Prune out-of-date logic to resolve default object_store_memory size #60683
Copy link
Copy link
Open
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalcoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray Core
Description
The PR to unify resource isolation config creation has exposed that the current logic to resolve the default object_store_memory contains many old artifacts that needs to be cleaned up.
This issue will track the following laundry list of items that needs to be improved:
- memory resolution logic is currently fragmented across many different function calls (e.g. resource_and_label_spec for memory resolution, determine_plasma_store_config for object store resolution). For clarity and consistency, memory resolution should be organized into a single location. (Refactoring the monolithic resolution function into multiple parts will also improve readability: [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment)).
- There was an issue with specific macbook many years ago, that we've created an adhoc solution for. We should create a more general fix for this: [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment).
- A large amount of memory was reserved to resolve a rounding error. This is a waste of memory on large machines: [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment).
- Code specific to platform repeated uses if statements to check what OS they are running on. We could create a single utility that runs code based on OS for cleanliness.
- [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment)
We will need to do a pass over these logic to unify and update the object_store resolution process. - We only cap the default object store memory with the object store memory cap when the user did not specify a custom value. This eliminates the point of having a cap, and we should also cap user specified object store size as well. [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment)
- After the PR to unify resource isolation config creation,
self.available_memoryandself.object_store_memoryshould only be None on connect only case. It will be more safe to make this point explicit: [Core] (Resource Isolation 1/n) Unify resource isolation config construction into a single step #59372 (comment)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalcoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray Core
Type
Projects
Status
Todo