Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently DS tasks can only run on worker servers, but in actual business scenarios, we often need to send tasks to remote server for execution, such as Shell Task.
So i suggest that could DS add a Host manage model under Security menu, such as Environment manage model. Then user can select the remote host to run the task.

📝 According to the communication at the bi-weekly meeting, the update plan is as follows：

• Add table named t_ds_task_remote_server in DB.
  

• Add TaskRemoteServer entity in dolphinscheduler-dao model.

• Add a task remote server manage page in Secerity menu, such as create, delete, edit and test connect.
  

• Add a task remote server select input (field: taskRemoteServerCode) in Shell and Python task form.

• Add task_remote_server_code column in t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log tables.

• Add taskRemoteServerCode field in TaskInstance, TaskDefintion, TaskDefinitionLog, TaskNode, and add taskRemoteServerInfo field in TaskInstance (not a table field).

• Add taskRemoteServerInfo(just contain ip, user, password, name) entity in dolphinscheduler-task-plugin model. And add taskRemoteServerInfofield in TaskExecutionContext.

• Shell and Python task will check TaskExecutionContext.getTaskRemoteServerInfo(), if not null, it will scp the command files and resource files to the task remote server, and send the start command to remote to exec it. If other task plugin need this feture, can also check this task server field.

• Use JSch 3rd to ssh and scp (scp resource files and command files to remote server).

• The task will ssh and scp to the task server as the DS running user, not tenant.

1. How to avoid causing IO overload?
The IO load will be protected by the following measures:

1. When the IO usage of Wroker is high, the current task will be not executed.
2. Set the total size threshold for single-task transfer files.
3. Control transfer rate.
4. Monitor the file transfer process, if the IO usage is too high, the transfer process will be terminated and the task will be set to fail.

2. Why do not use remote jar and let remote server download the resource files from resource center?
I think this will increase the complexity of the remote server's environment, because the remote server is not part of the DS cluster. For example, the remote user need have the permission to download the resource files, and need install JRE.

Use case

1. User can manage the hosts in Security menu.
2. When config the task node, the user can select the remote host to run this task.

Related issues

N/A

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct