Skip to content

Use unified memory management for Comet all the times #1017

@viirya

Description

@viirya

What is the problem the feature request solves?

We currently only enable unified memory manager for Comet if off-heap mode is enabled. It causes some memory issues like #996, #949, etc.

Re-thinking about this, I think a better approach is to always use unified memory management for Comet, i.e. off-heap mode must be enabled for Comet. It could simplify memory management in Comet. It can give a more explicit signal to users that off-heap memory is used for Comet native execution that should give users a more clear picture about how Spark/Comet memory is used (so they don't need to guess).

Describe the potential solution

No response

Additional context

No response

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions