Action List: Extension of operations for the k8s YAML task:

• Pod [DSIP-66][k8s] YAML Pod for User Customization #16482

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Supporting user-customized K8s YAML tasks has the following benefits:

• Flexibility: Unlike the existing K8s low-code job with limited functionality, YAML tasks provide users with the flexibility to define sophisticated task instances in DolphinScheduler, similar to how custom JSON does in DataX.

• Workflow Customization: Users can integrate operational and maintenance processes into DolphinScheduler using YAML for complex workflows.

• Configuration Requirements: The current K8s low-code job does not meet users' in-depth needs, particularly for tasks involving multiple pods or specific configurations like environment variables and tolerations; in contrast, K8s YAML tasks do.

In short, by enabling user-customized YAML tasks, DolphinScheduler can better support a wide range of Kubernetes-based workflows and operational requirements.

Design Detail

2.1 Design Overview

The following is a Swimlane Diagram showing how this k8s YAML task is embedded into Apache DolphinScheduler:

Figure 2-1(1). Design Overview

1. User starts a Web page to edit and save K8s YAML Workflow.
2. UI provides an editor for user to input YAML in Custom Template mode.
3. API Server encapsulates command and hands it over to Master.
4. Master splits the workflow DAG and dispatches tasks to Worker.
5. Worker picks the appropriate task executor and operation. E.g., for k8s Pod YAML, Worker picks YAML Task Executor, and then picks Pod Operation.
6. Worker reports status to Master.
7. User reviews k8s YAML task log in the Task Instance Window.

2.2 Frontend Design

The frontend adds support for user-customized k8s YAML tasks while remaining compatible with the original k8s low-code jobs.

Figure 2-2(1). Design Overview

1. The Web UI layouts

   When the user switches on the Custom Template, the Low-code k8s Job fields should hide and YAML editor should appear (or vice versa), similar to the JSON Custom Template in the DataX plugin.

   This feature, as shown in Figure 2-2(1), is implemented using the Vue component span, which is controlled by reactive variables (such as yamlEditorSpan) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

2. The Request body

   When the user switches to Custom Template mode, the request body should include only YAML-related fields (customConfig and yamlContent), and all previously hidden fields should not be sent.

   This feature is implemented using the taskParams in the file dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts

3. i18n/locales

   Apache DolphinScheduler is an international software and should support multiple languages.

   The text on the Web UI are retrieved from variables defined in the file dolphinscheduler-ui/src/locales/{en_US, zh_CN}/project.ts. And for user-customized k8s YAML tasks, there are three key variables to consider:

   • k8s_custom_template: the label for the switch to enable user-customized k8s YAML tasks.
   • k8s_yaml_template: the label for the text editor used to input user YAML.
   • k8s_yaml_empty_tips: the warning message displayed when a user tries to submit empty YAML

   This feature is implemented by invoking t('project.node.${variable_name}') (such as t('project.node.k8s_yaml_template')) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

2.3 Backend Design

The backend design describes the process of how the worker executes user-customized k8s YAML tasks. As shown in Figure 2-3(1), we can see how user-customized k8s YAML Pod tasks are related to the original k8s low-code jobs.

Figure 2-3(1). Backend Design Overview

After the worker checks the parameters, K8sYamlTaskExecutor is loaded for the current user-customized k8s YAML Pod task. Once the YAML is parsed into HasMetadata, its kind field is used to assign abstractK8sOperation as K8sPodOperation for executing the YAML Pod task.

1. K8s Task Executors

   
   Figure 2-3(2). K8s Task Executors

   Three k8s task executor are involved, as shown in Figure 2-3(2):

   • AbstractK8sTaskExecutor is an abstract class that represents a k8s task executor.
   • K8sTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a low-code executor
   • K8sYamlTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a user-customized k8s YAML task executor.

2. K8s Operation handler

   
   Figure 2-3(3). K8s Operation Handlers

   Two operation handlers are involved, as shown in Figure 2-3(3):

   • AbstractK8sOperation is an interface representing all k8s resource operations.
   • K8sPodOperation is a concrete class that implements AbstractK8sOperation to handle Pod operations

2.4 Usecase Design

A typical use case for a k8s YAML task includes uploading YAML, online workflows, and starting workflows, similar to k8s low-code jobs, unless users switch to the Custom Template option to fill in YAML.

Figure 2-4(1). Usecase Design

1. The user edits a k8s YAML node in a workflow
2. If the Custom Template is activated and YAML content is not blank, the user may online this whole workflow
3. If the workflow is online, the user may start the workflow and review the logs generated during the execution of the workflow.

Compatibility, Deprecation, and Migration Plan

3.1 Compatibility Plan

The user-customized k8s YAML feature requires only customConfig to be activated, By default, the value is 0, which applies to the existing k8s low-code jobs.

The remainder of this section will demonstrate the flexibility and compatibility of this design by using the example of introducing Configmaps:

    this.k8sYamlType = K8sYamlType.valueOf(this.metadata.getKind());
    generateOperation();

After parsing with YamlUtils::load, the kind field acquired by this.metadata.getKind() will be ConfigMaps. Then, this.k8sYamlType is determined and used to generate the corresponding operations:

    private void generateOperation() {
        switch (k8sYamlType) {
            case Pod:
                abstractK8sOperation = new K8sPodOperation(k8sUtils.getClient());
                break;
            case ConfigMaps:
                abstractK8sOperation = new K8sConfigmapsOperation(k8sUtils.getClient());
                break;
            default:
                throw new TaskException(
                        String.format("K8sYamlTaskExecutor do not support type %s", k8sYamlType.name()));
        }
    }

Consequently, generateOperation() will set this.abstractK8sOperation to a new instance of K8sConfigmapsOperation. Next, we can implement K8sConfigmapsOperation to handle the ConfigMaps operations.

3.2 Deprecation Plan

N/A for now, waiting for community opinions.

3.3 Migration Plan

N/A for now, waiting for community opinions.

Test Plan

4.1 Overview

The User-customized k8s YAML task feature allows users to submit YAML task to k8s, including Pod, ConfigMaps, and other resources.

This test plan aims to ensure that the feature functions as expected and meets user requirements.

4.2 Scope

1. YAML Pod

Code of Conduct

• I agree to follow this project's Code of Conduct