Reduce the size of tarball to continue ASF release by zhongjiajie · Pull Request #15004 · apache/dolphinscheduler

zhongjiajie · 2023-10-10T11:21:07Z

the reason can see in https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt

the major change in this patch is:

Add module dolphinscheduler-task-all-prune only contain artifactId starting with dolphinscheduler-task- to reduce the third part library from task plugin, cause service api-server and master-server only need dolphinscheduler-task- to verify parameter(api-server), get local parameter and resource(master-server), so it does not need others third-party dependence
Remove some not use dependencies in api-server and master-server.
Module tools dynamic load libs from other servers in binary tarball.

the binary tarball size is reduced to ASF limit 950MB

- -rw-r--r-- 1.2G apache-dolphinscheduler-dev-SNAPSHOT-bin.tar.gz
+ -rw-r--r-- 860M apache-dolphinscheduler-dev-SNAPSHOT-bin.tar.gz

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt

zhongjiajie · 2023-10-10T11:26:28Z

dolphinscheduler-task-plugin/dolphinscheduler-task-all-prune/pom.xml

+        <version>dev-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>dolphinscheduler-task-all-prune</artifactId>


not sure whether exists a better way to do that or not

<dependency> <groupId>org.apache.dolphinscheduler</groupId> <artifactId>dolphinscheduler-task-dataquality</artifactId> <version>${project.version}</version> <exclusions> <exclusion> <groupId>*</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency>

we exclude all libs but only keep dolphinscheduler-task-* themself

zhongjiajie · 2023-10-10T11:27:48Z

...uler-master/src/main/java/org/apache/dolphinscheduler/server/master/metrics/TaskMetrics.java

 import lombok.experimental.UtilityClass;

-import com.facebook.presto.jdbc.internal.guava.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSet;


remove dependence module dolphinscheduler-task-all and some of lib not exists anymore. so we have to change some code

zhongjiajie · 2023-10-10T11:29:54Z

dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml

+                <exclude>libs/</exclude>
+                <exclude>bin/</exclude>
+                <exclude>dist-bin/</exclude>
+            </excludes>
        </fileSet>
-
        <fileSet>
-            <directory>${basedir}/../dolphinscheduler-dist/target/dolphinscheduler-dist-${project.version}</directory>
-            <outputDirectory>.</outputDirectory>
+            <directory>${basedir}/../dolphinscheduler-tools/target/tools/libs</directory>
+            <outputDirectory>tools/libs</outputDirectory>
+            <includes>
+                <include>dolphinscheduler-*.jar</include>
+                <include>spring-*.jar</include>
+            </includes>
+        </fileSet>
+        <fileSet>
+            <directory>${basedir}/../dolphinscheduler-tools/target/tools/dist-bin</directory>
+            <outputDirectory>tools/bin</outputDirectory>


In binary package module tools, we only need some spring jar and use dir dist-bin to load libs from others server

zhongjiajie · 2023-10-10T11:31:11Z

dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml

+            <directory>${basedir}/../dolphinscheduler-dist/target/dolphinscheduler-dist-${project.version}</directory>
+            <outputDirectory>.</outputDirectory>


remove ui directory in the root level of package, cause we already have the most use one under api-server

zhongjiajie · 2023-10-10T11:31:36Z

dolphinscheduler-master/pom.xml

-
-        <dependency>
-            <groupId>org.apache.dolphinscheduler</groupId>
-            <artifactId>dolphinscheduler-storage-all</artifactId>


master do not need storage module

zhongjiajie · 2023-10-10T11:34:28Z

dolphinscheduler-tools/src/main/dist-bin/migrate-resource.sh

+CP=$DOLPHINSCHEDULER_HOME/tools/libs/*
+for d in api-server; do
+  for f in $DOLPHINSCHEDULER_HOME/$d/libs/*.jar; do
+    JAR_FILE_NAME=${f##*/}
+    if [[ ! $CP =~ $JAR_FILE_NAME ]] && [[ ! $JAR_FILE_NAME =~ "dolphinscheduler-" ]] && [[ ! $JAR_FILE_NAME == "spring"* ]]; then
+      CP=$CP:$f
+    fi
+  done
+done


load all libs in api-server exclusion dolphinscheduler-* and ^spring* to avoid launch service from api-server

codecov-commenter · 2023-10-10T11:45:31Z

Codecov Report

❗ No coverage uploaded for pull request base (3.2.0-release@ab306ce). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 27c9d15 differs from pull request most recent head 4dfdb0d. Consider uploading reports for the commit 4dfdb0d to get more accurate results

@@               Coverage Diff                @@
##             3.2.0-release   #15004   +/-   ##
================================================
  Coverage                 ?   38.84%           
  Complexity               ?     4683           
================================================
  Files                    ?     1289           
  Lines                    ?    44345           
  Branches                 ?     4885           
================================================
  Hits                     ?    17225           
  Misses                   ?    25218           
  Partials                 ?     1902

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

caishunfeng

LGTM

caishunfeng · 2023-10-10T12:14:52Z

Need to check docker and k8s way.

SbloodyS

That's cool~

ruanwenjun · 2023-10-10T12:55:09Z

We need to modify the license file.

zhongjiajie · 2023-10-10T13:23:27Z

Need to check docker and k8s way.

@caishunfeng could you help check k8s?

zhongjiajie · 2023-10-10T13:50:29Z

Need to check docker and k8s way.

Seems we already have K8S test in our CI

sonarqubecloud · 2023-10-11T08:47:22Z

SonarCloud Quality Gate failed.

8 Bugs
0 Vulnerabilities
1 Security Hotspot
111 Code Smells

22.9% Coverage
1.3% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

* Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt

…apache#15540) * Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt # Conflicts: # tools/dependencies/known-dependencies.txt

…ASF release (#16042) * add version of 3.2.2 * Fix rpc channel leak due to concurrent operation (#16021) * Fix rpc channel leak due to concurrent operation * Throw channel create failed exception --------- Co-authored-by: Rick Cheng <rickchengx@gmail.com> * Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (#16027) * cp: Reduce the size of tarball to continue ASF release (#15004) (#15540) * Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt # Conflicts: # tools/dependencies/known-dependencies.txt * Increase block-until-connected in ZookeeperRegistryTestCase (#16041) * [Fix][CI] fix the ci error of Values.datasource.profile (#16031) Co-authored-by: Eric Gao <ericgao.apache@gmail.com> * Correction of Typos in the Chinese Document Appendix for Task Parameters (#16033) Co-authored-by: Rick Cheng <rickchengx@gmail.com> * [DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type (#15967) * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * improvement project code * remove useless imports * remove method onClearSearchTaskType * add parameter doc * optimisation logic * code conflict resolution * code conflict resolution * [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (#15978) * update * test * add monitor enhance ui * update * update * update doc * fix spotless * update * update * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <wenjun@apache.org> * update * fix spotless * update --------- Co-authored-by: Wenjun Ruan <wenjun@apache.org> * [Improvement][Monitor] Add UT for montor (#15998) * formatting Code * add pom * Fix NettyRemotingClient might throw IllegalMonitorStateException (#16038) * Removed unused StateEventHandleFailure (#16052) * Remove OmitStackTraceInFastThrow in start.sh (#16054) * Update Chart.yaml Rollback APPversion * modify log msg (#16062) Co-authored-by: xiangzihao <460888207@qq.com> * [DS-16046][fix] Set PreparedStatement parameter type (#16050) Fix the PreparedStatement parameter is TIME, set it to java.sql.Time Co-authored-by: Aaron Wang <wangweirao16@gmail.com> Co-authored-by: Rick Cheng <rickchengx@gmail.com> * Update pom.xml * [DSIP-42] Add dolphinscheduler-aws-authentication module (#16043) --------- Co-authored-by: wangxj <wangxj31> Co-authored-by: Wenjun Ruan <wenjun@apache.org> Co-authored-by: Rick Cheng <rickchengx@gmail.com> Co-authored-by: Eric Gao <ericgao.apache@gmail.com> Co-authored-by: TianXinCoord <tianxincoord@163.com> Co-authored-by: 小可耐 <46134044+sdhzwc@users.noreply.github.com> Co-authored-by: sleo <97011595+alei1206@users.noreply.github.com> Co-authored-by: xiangzihao <460888207@qq.com> Co-authored-by: yinxiaolog <43245392+yinxiaolog@users.noreply.github.com> Co-authored-by: Aaron Wang <wangweirao16@gmail.com>

Reduce the size of tarball to continue ASF release

bd8a5d5

for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt

zhongjiajie requested review from EricGao888, SbloodyS, caishunfeng, ruanwenjun and zhuangchong as code owners October 10, 2023 11:21

github-actions bot added the backend label Oct 10, 2023

zhongjiajie added 2 commits October 10, 2023 19:22

rename the new module

0a79ce4

spotless

7366a15

zhongjiajie commented Oct 10, 2023

View reviewed changes

caishunfeng approved these changes Oct 10, 2023

View reviewed changes

SbloodyS approved these changes Oct 10, 2023

View reviewed changes

license

bf31ee9

fix api test

85dc6f4

github-actions bot added the test label Oct 11, 2023

fix api test

4dfdb0d

zhongjiajie added this to the 3.2.0 milestone Oct 11, 2023

zhongjiajie force-pushed the 3.2.0-release branch from a1cf67e to a8272b2 Compare October 11, 2023 08:05

Merge branch '3.2.0-release' into reduce-tarball-size

115ac92

zhongjiajie merged commit 6d7dbe6 into apache:3.2.0-release Oct 11, 2023

zhongjiajie deleted the reduce-tarball-size branch October 11, 2023 08:19

		<directory>${basedir}/../dolphinscheduler-dist/target/dolphinscheduler-dist-${project.version}</directory>
		<outputDirectory>.</outputDirectory>

Conversation

zhongjiajie commented Oct 10, 2023

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

caishunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

caishunfeng commented Oct 10, 2023

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

ruanwenjun commented Oct 10, 2023

Uh oh!

zhongjiajie commented Oct 10, 2023

Uh oh!

zhongjiajie commented Oct 10, 2023

Uh oh!

sonarqubecloud bot commented Oct 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Oct 10, 2023 •

edited

Loading