Skip to content

Add actuator and enable metrics, add Grafana configs#6840

Merged
CalvinKirs merged 1 commit intoapache:devfrom
kezhenxu94:metrics
Nov 14, 2021
Merged

Add actuator and enable metrics, add Grafana configs#6840
CalvinKirs merged 1 commit intoapache:devfrom
kezhenxu94:metrics

Conversation

@kezhenxu94
Copy link
Copy Markdown
Member

Also closes #6520

PTAL @CalvinKirs @dailidong @lenboo @caishunfeng

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 13, 2021

Codecov Report

Merging #6840 (056a491) into dev (e76cf77) will decrease coverage by 8.65%.
The diff coverage is 18.75%.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #6840      +/-   ##
============================================
- Coverage     41.80%   33.15%   -8.66%     
+ Complexity     3614     1620    -1994     
============================================
  Files           641      433     -208     
  Lines         25891    14300   -11591     
  Branches       2796     1428    -1368     
============================================
- Hits          10825     4741    -6084     
+ Misses        14087     9107    -4980     
+ Partials        979      452     -527     
Impacted Files Coverage Δ
.../org/apache/dolphinscheduler/common/Constants.java 79.16% <ø> (ø)
...he/dolphinscheduler/common/thread/ThreadUtils.java 0.00% <ø> (ø)
.../datasource/api/client/CommonDataSourceClient.java 0.00% <0.00%> (ø)
...r/plugin/datasource/hive/HiveDataSourceClient.java 3.03% <ø> (+0.17%) ⬆️
...e/dolphinscheduler/server/master/MasterServer.java 0.00% <0.00%> (ø)
...ver/master/consumer/TaskPriorityQueueConsumer.java 0.00% <0.00%> (ø)
...server/master/dispatch/host/HostManagerConfig.java 0.00% <0.00%> (ø)
...uler/server/master/runner/EventExecuteService.java 0.00% <0.00%> (ø)
...r/server/master/runner/MasterSchedulerService.java 0.00% <0.00%> (ø)
...er/server/master/runner/WorkflowExecuteThread.java 11.31% <0.00%> (ø)
... and 234 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e76cf77...056a491. Read the comment docs.

@sonarqubecloud
Copy link
Copy Markdown

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 5 Code Smells

12.1% 12.1% Coverage
0.0% 0.0% Duplication

Copy link
Copy Markdown
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CalvinKirs CalvinKirs added the feature new feature label Nov 14, 2021
@CalvinKirs CalvinKirs added this to the 2.0.1-release milestone Nov 14, 2021
@CalvinKirs CalvinKirs merged commit 4114cb0 into apache:dev Nov 14, 2021
@kezhenxu94 kezhenxu94 deleted the metrics branch November 14, 2021 12:35
Comment on lines -24 to -78
# 1.replace file
echo "1.replace file"

txt=""
if [[ "$OSTYPE" == "darwin"* ]]; then
# Mac OSX
txt="''"
fi

datasourceDriverClassname="com.mysql.jdbc.Driver"
if [ $dbtype == "postgresql" ];then
datasourceDriverClassname="org.postgresql.Driver"
fi

# Change configuration in conf/config/dolphinscheduler_env.sh
sed -i ${txt} "s@^export JAVA_HOME=.*@export JAVA_HOME=${javaHome}@g" conf/env/dolphinscheduler_env.sh

# Change configuration in conf/datasource.properties
sed -i ${txt} "s@^spring.datasource.driver-class-name=.*@spring.datasource.driver-class-name=${datasourceDriverClassname}@g" conf/datasource.properties
sed -i ${txt} "s@^spring.datasource.url=.*@spring.datasource.url=jdbc:${dbtype}://${dbhost}/${dbname}?characterEncoding=UTF-8\&allowMultiQueries=true@g" conf/datasource.properties
sed -i ${txt} "s@^spring.datasource.username=.*@spring.datasource.username=${username}@g" conf/datasource.properties
sed -i ${txt} "s@^spring.datasource.password=.*@spring.datasource.password=${password}@g" conf/datasource.properties

# Change configuration in conf/common.properties
sed -i ${txt} "s@^data.basedir.path=.*@data.basedir.path=${dataBasedirPath}@g" conf/common.properties
sed -i ${txt} "s@^resource.storage.type=.*@resource.storage.type=${resourceStorageType}@g" conf/common.properties
sed -i ${txt} "s@^resource.upload.path=.*@resource.upload.path=${resourceUploadPath}@g" conf/common.properties
sed -i ${txt} "s@^hadoop.security.authentication.startup.state=.*@hadoop.security.authentication.startup.state=${kerberosStartUp}@g" conf/common.properties
sed -i ${txt} "s@^java.security.krb5.conf.path=.*@java.security.krb5.conf.path=${krb5ConfPath}@g" conf/common.properties
sed -i ${txt} "s@^login.user.keytab.username=.*@login.user.keytab.username=${keytabUserName}@g" conf/common.properties
sed -i ${txt} "s@^login.user.keytab.path=.*@login.user.keytab.path=${keytabPath}@g" conf/common.properties
sed -i ${txt} "s@^kerberos.expire.time=.*@kerberos.expire.time=${kerberosExpireTime}@g" conf/common.properties
sed -i ${txt} "s@^hdfs.root.user=.*@hdfs.root.user=${hdfsRootUser}@g" conf/common.properties
sed -i ${txt} "s@^fs.defaultFS=.*@fs.defaultFS=${defaultFS}@g" conf/common.properties
sed -i ${txt} "s@^fs.s3a.endpoint=.*@fs.s3a.endpoint=${s3Endpoint}@g" conf/common.properties
sed -i ${txt} "s@^fs.s3a.access.key=.*@fs.s3a.access.key=${s3AccessKey}@g" conf/common.properties
sed -i ${txt} "s@^fs.s3a.secret.key=.*@fs.s3a.secret.key=${s3SecretKey}@g" conf/common.properties
sed -i ${txt} "s@^resource.manager.httpaddress.port=.*@resource.manager.httpaddress.port=${resourceManagerHttpAddressPort}@g" conf/common.properties
sed -i ${txt} "s@^yarn.resourcemanager.ha.rm.ids=.*@yarn.resourcemanager.ha.rm.ids=${yarnHaIps}@g" conf/common.properties
sed -i ${txt} "s@^yarn.application.status.address=.*@yarn.application.status.address=http://${singleYarnIp}:%s/ws/v1/cluster/apps/%s@g" conf/common.properties
sed -i ${txt} "s@^yarn.job.history.status.address=.*@yarn.job.history.status.address=http://${singleYarnIp}:19888/ws/v1/history/mapreduce/jobs/%s@g" conf/common.properties
sed -i ${txt} "s@^sudo.enable=.*@sudo.enable=${sudoEnable}@g" conf/common.properties

# The following configurations may be commented, so ddd #* to ensure sed work correct
# Change configuration in conf/worker.properties
sed -i ${txt} "s@^#*worker.tenant.auto.create=.*@worker.tenant.auto.create=${workerTenantAutoCreate}@g" conf/worker.properties
sed -i ${txt} "s@^#*alert.listen.host=.*@alert.listen.host=${alertServer}@g" conf/worker.properties

# Change configuration in conf/application-api.properties
sed -i ${txt} "s@^#*server.port=.*@server.port=${apiServerPort}@g" conf/application-api.properties

# Change configuration in conf/registry.properties
sed -i ${txt} "s@^#*registry.plugin.dir=.*@registry.plugin.dir=${installPath}/${registryPluginDir}@g" conf/registry.properties
sed -i ${txt} "s@^#*registry.plugin.name=.*@registry.plugin.name=${registryPluginName}@g" conf/registry.properties
sed -i ${txt} "s@^#*registry.servers=.*@registry.servers=${registryServers}@g" conf/registry.properties
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kezhenxu94, why should we remove those lines?

@chengshiwen
Copy link
Copy Markdown
Member

chengshiwen commented Nov 25, 2021

This pr violates the principle of minimum code submission, introduces irrelevant modifications and deletes template files related to docker, resulting in docker and k8s not working properly.
image

@chengshiwen
Copy link
Copy Markdown
Member

Maybe related issue: #6998, #6994, #6885
@CalvinKirs @kezhenxu94

@kezhenxu94
Copy link
Copy Markdown
Member Author

Maybe related issue: #6998, #6994, #6885

@CalvinKirs @kezhenxu94

Only #6998 is related to this as this only lands in dev branch and the other two issues are 2.0.0 and 1.3.9. At this moment dev is not expected to be used in production environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] configration of grafana

5 participants