Skip to content

Add Kubernetes configmap reload to all components#11730

Merged
kezhenxu94 merged 1 commit intodevfrom
config-reload
Sep 2, 2022
Merged

Add Kubernetes configmap reload to all components#11730
kezhenxu94 merged 1 commit intodevfrom
config-reload

Conversation

@kezhenxu94
Copy link
Copy Markdown
Member

@kezhenxu94 kezhenxu94 commented Sep 1, 2022 

[INFO] 2022-09-01 07:15:00.365 +0000 org.apache.dolphinscheduler.server.worker.task.WorkerHeartBeatTask:[86] - [WorkflowInstance-0][TaskInstance-0] - Success write worker group heartBeatInfo into registry, workGroupPath: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234] workerHeartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016500362,"cpuUsage":0.04,"memoryUsage":0.0,"loadAverage":0.17,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"serverStatus":0,"processId":8,"workerHostWeight":100,"workerWaitingTaskCount":0,"workerExecThreadCount":10}
[WARN] 2022-09-01 07:15:03.775 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-kubernetes' not present in namespace : 'default'
[WARN] 2022-09-01 07:15:03.780 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-h2' not present in namespace : 'default'
[INFO] 2022-09-01 07:15:03.781 +0000 org.springframework.cloud.kubernetes.fabric8.config.reload.EventBasedConfigMapChangeDetector:[137] - [WorkflowInstance-0][TaskInstance-0] - Detected change in config maps
[INFO] 2022-09-01 07:15:03.781 +0000 org.springframework.cloud.kubernetes.fabric8.config.reload.EventBasedConfigMapChangeDetector:[61] - [WorkflowInstance-0][TaskInstance-0] - Reloading using strategy: REFRESH
[WARN] 2022-09-01 07:15:04.003 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-kubernetes' not present in namespace : 'default'
[WARN] 2022-09-01 07:15:04.010 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-h2' not present in namespace : 'default'
[INFO] 2022-09-01 07:15:04.011 +0000 org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration:[109] - [WorkflowInstance-0][TaskInstance-0] - Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.standalone-server.default'}]
[INFO] 2022-09-01 07:15:04.017 +0000 org.springframework.boot.SpringApplication:[640] - [WorkflowInstance-0][TaskInstance-0] - The following 2 profiles are active: "kubernetes", "h2"
[INFO] 2022-09-01 07:15:04.050 +0000 org.springframework.boot.SpringApplication:[61] - [WorkflowInstance-0][TaskInstance-0] - Started application in 0.261 seconds (JVM running for 152.919)
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[147] - [WorkflowInstance-0][TaskInstance-0] - Master config: listenPort -> 5678 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[148] - [WorkflowInstance-0][TaskInstance-0] - Master config: fetchCommandNum -> 10 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[149] - [WorkflowInstance-0][TaskInstance-0] - Master config: preExecThreads -> 10 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[150] - [WorkflowInstance-0][TaskInstance-0] - Master config: execThreads -> 10 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[151] - [WorkflowInstance-0][TaskInstance-0] - Master config: dispatchTaskNumber -> 3 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[152] - [WorkflowInstance-0][TaskInstance-0] - Master config: hostSelector -> LOWER_WEIGHT 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[153] - [WorkflowInstance-0][TaskInstance-0] - Master config: heartbeatInterval -> PT10S 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[154] - [WorkflowInstance-0][TaskInstance-0] - Master config: taskCommitRetryTimes -> 5 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[155] - [WorkflowInstance-0][TaskInstance-0] - Master config: taskCommitInterval -> PT1S 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[156] - [WorkflowInstance-0][TaskInstance-0] - Master config: stateWheelInterval -> PT5S 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[157] - [WorkflowInstance-0][TaskInstance-0] - Master config: maxCpuLoadAvg -> 8.0 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[158] - [WorkflowInstance-0][TaskInstance-0] - Master config: reservedMemory -> 0.3 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[159] - [WorkflowInstance-0][TaskInstance-0] - Master config: failoverInterval -> PT10M 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[160] - [WorkflowInstance-0][TaskInstance-0] - Master config: killYarnJobWhenTaskFailover -> true 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[161] - [WorkflowInstance-0][TaskInstance-0] - Master config: registryDisconnectStrategy -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S) 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[162] - [WorkflowInstance-0][TaskInstance-0] - Master config: masterAddress -> demo-deployment-75b5f8457-ccnck:5678 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[163] - [WorkflowInstance-0][TaskInstance-0] - Master config: masterRegistryPath -> /nodes/master/demo-deployment-75b5f8457-ccnck:5678 
[INFO] 2022-09-01 07:15:04.254 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[100] - [WorkflowInstance-0][TaskInstance-0] - Worker config: listenPort -> 1234
[INFO] 2022-09-01 07:15:04.255 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[101] - [WorkflowInstance-0][TaskInstance-0] - Worker config: execThreads -> 20
[INFO] 2022-09-01 07:15:04.256 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[102] - [WorkflowInstance-0][TaskInstance-0] - Worker config: heartbeatInterval -> PT10S
[INFO] 2022-09-01 07:15:04.257 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[103] - [WorkflowInstance-0][TaskInstance-0] - Worker config: hostWeight -> 100
[INFO] 2022-09-01 07:15:04.258 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[104] - [WorkflowInstance-0][TaskInstance-0] - Worker config: tenantAutoCreate -> true
[INFO] 2022-09-01 07:15:04.259 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[105] - [WorkflowInstance-0][TaskInstance-0] - Worker config: tenantDistributedUser -> false
[INFO] 2022-09-01 07:15:04.259 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[106] - [WorkflowInstance-0][TaskInstance-0] - Worker config: maxCpuLoadAvg -> 8
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[107] - [WorkflowInstance-0][TaskInstance-0] - Worker config: reservedMemory -> 0.3
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[108] - [WorkflowInstance-0][TaskInstance-0] - Worker config: groups -> [default]
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[109] - [WorkflowInstance-0][TaskInstance-0] - Worker config: alertListenHost -> localhost
[INFO] 2022-09-01 07:15:04.261 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[110] - [WorkflowInstance-0][TaskInstance-0] - Worker config: alertListenPort -> 50052
[INFO] 2022-09-01 07:15:04.261 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[111] - [WorkflowInstance-0][TaskInstance-0] - Worker config: registryDisconnectStrategy -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S)
[INFO] 2022-09-01 07:15:04.262 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[112] - [WorkflowInstance-0][TaskInstance-0] - Worker config: workerAddress -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S)
[INFO] 2022-09-01 07:15:04.262 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[113] - [WorkflowInstance-0][TaskInstance-0] - Worker config: workerGroupRegistryPaths: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234]
[INFO] 2022-09-01 07:15:09.042 +0000 org.apache.dolphinscheduler.server.master.task.MasterHeartBeatTask:[68] - [WorkflowInstance-0][TaskInstance-0] - Success write master heartBeatInfo into registry, masterRegistryPath: /nodes/master/demo-deployment-75b5f8457-ccnck:5678, heartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016509037,"cpuUsage":0.08,"memoryUsage":0.0,"loadAverage":0.16,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"processId":8}
[INFO] 2022-09-01 07:15:10.371 +0000 org.apache.dolphinscheduler.server.worker.task.WorkerHeartBeatTask:[86] - [WorkflowInstance-0][TaskInstance-0] - Success write worker group heartBeatInfo into registry, workGroupPath: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234] workerHeartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016510366,"cpuUsage":0.02,"memoryUsage":0.0,"loadAverage":0.14,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"serverStatus":0,"processId":8,"workerHostWeight":100,"workerWaitingTaskCount":0,"workerExecThreadCount":20}

As we can see Detected change in config maps and Reloading using strategy: REFRESH, and from the log workerExecThreadCount":10 to workerExecThreadCount":20 when I modify the config map to change the workerExecThreadCount from 10 to 20.

Now all these work out of box in Kubernetes.

@kezhenxu94
Copy link
Copy Markdown
Member Author

kezhenxu94 commented Sep 1, 2022

All works as before, when users want to change the configuration, just create a patch config like this, and it's all done.

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  application.yml: |-
    worker:
      exec-threads: 30
kind: ConfigMap
metadata:
  name: standalone-server
  namespace: default
EOF

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Sep 1, 2022 •
edited
Loading

Codecov Report

Merging #11730 (f88feec) into dev (d0f5e7e) will decrease coverage by 0.09%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev   #11730      +/-   ##
============================================
- Coverage     39.73%   39.63%   -0.10%     
+ Complexity     4709     4689      -20     
============================================
  Files          1008     1008              
  Lines         37878    37881       +3     
  Branches       4238     4238              
============================================
- Hits          15049    15016      -33     
- Misses        21218    21259      +41     
+ Partials       1611     1606       -5
Impacted Files Coverage Δ
...inscheduler/server/log/LoggerRequestProcessor.java 17.30% <0.00%> (-0.52%) ⬇️
...eduler/server/worker/task/WorkerHeartBeatTask.java 68.08% <0.00%> (-8.52%) ⬇️
...erver/master/processor/queue/TaskEventService.java 75.00% <0.00%> (-5.36%) ⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java 55.12% <0.00%> (-1.29%) ⬇️
...ache/dolphinscheduler/plugin/task/k8s/K8sTask.java 89.47% <0.00%> (ø)
...lphinscheduler/plugin/task/k8s/K8sTaskChannel.java 0.00% <0.00%> (ø)
...lphinscheduler/plugin/task/dq/DataQualityTask.java 0.00% <0.00%> (ø)
...heduler/plugin/task/dq/DataQualityTaskChannel.java 0.00% <0.00%> (ø)
...inscheduler/plugin/task/k8s/K8sTaskParameters.java
...ler/plugin/task/dq/utils/spark/SparkArgsUtils.java
... and 10 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

EricGao888
EricGao888 previously approved these changes Sep 2, 2022
View reviewed changes
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Sep 2, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@EricGao888 EricGao888 added this to the 3.1.0 milestone Sep 2, 2022
@kezhenxu94 kezhenxu94 merged commit 277f137 into dev Sep 2, 2022
@kezhenxu94 kezhenxu94 deleted the config-reload branch September 2, 2022 04:03
@kezhenxu94 kezhenxu94 mentioned this pull request Sep 5, 2022
Closed
@caishunfeng caishunfeng mentioned this pull request Sep 8, 2022
Closed
1 task
fengjian1129 pushed a commit to fengjian1129/dolphinscheduler that referenced this pull request Sep 13, 2022
@kezhenxu94
Add Kubernetes configmap reload to all components (apache#11730)
6869c50
fengjian1129 pushed a commit to fengjian1129/dolphinscheduler that referenced this pull request Sep 29, 2022
@kezhenxu94
Add Kubernetes configmap reload to all components (apache#11730)
a8c0073
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 13, 2022
@kezhenxu94 @xdu-chenrj
Add Kubernetes configmap reload to all components (apache#11730)
fbe6eda
@kezhenxu94 kezhenxu94 mentioned this pull request Oct 27, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@EricGao888 EricGao888 EricGao888 approved these changes

@caishunfeng caishunfeng Awaiting requested review from caishunfeng caishunfeng is a code owner

@SbloodyS SbloodyS Awaiting requested review from SbloodyS SbloodyS is a code owner

@ruanwenjun ruanwenjun Awaiting requested review from ruanwenjun ruanwenjun is a code owner

@zhongjiajie zhongjiajie Awaiting requested review from zhongjiajie

@Tianqi-Dotes Tianqi-Dotes Awaiting requested review from Tianqi-Dotes

Assignees

@kezhenxu94 kezhenxu94

Labels

backend document

Projects

None yet

Milestone

3.1.0

Development

Successfully merging this pull request may close these issues.

3 participants

@kezhenxu94 @codecov-commenter @EricGao888