Skip to content

[Feature][master] When deployed on k8s and yarn, dolphinscheduler supports worker elastic scaling #9337

@huagetai

Description

@huagetai

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

When deployed on cluster managers such as k8s and yarn, dolphinscheduler supports worker elastic scaling
The initial idea is as follows:
DolphinSchedule 001

  1. Add ResourceManager to the master server.
  2. ResourceManager is responsible for the management and application of worker resources.
  3. ResourceManager perceives the online and offline of worker nodes through the register.
  4. The ResourceManager goes to the cluster manager to apply for resources according to the per workflow. The resources are released at the end of the workflow.

Use case

worker elastic scaling

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions