[Feature][DataQuality] Add data quality task backend by zixi0825 · Pull Request #4883 · apache/dolphinscheduler

zixi0825 · 2021-02-26T03:54:32Z

What is the purpose of the pull request

This pull request adds enum、rule relevant entity and task parameter that are used in data quality task

codecov-io · 2021-02-26T04:08:55Z

Codecov Report

Merging #4883 (9db18f8) into data_quality_design (14d49bc) will decrease coverage by 0.20%.
The diff coverage is 41.02%.

@@                    Coverage Diff                    @@
##             data_quality_design    #4883      +/-   ##
=========================================================
- Coverage                  45.01%   44.80%   -0.21%     
- Complexity                  3376     3557     +181     
=========================================================
  Files                        537      581      +44     
  Lines                      23077    24555    +1478     
  Branches                    2691     2801     +110     
=========================================================
+ Hits                       10387    11001     +614     
- Misses                     11762    12589     +827     
- Partials                     928      965      +37

Impacted Files	Coverage Δ	Complexity Δ
.../org/apache/dolphinscheduler/common/Constants.java	`85.00% <ø> (ø)`	`1.00 <0.00> (ø)`
...g/apache/dolphinscheduler/common/enums/DbType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...he/dolphinscheduler/common/enums/dq/CheckType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...olphinscheduler/common/enums/dq/ConnectorType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...inscheduler/common/enums/dq/DqFailureStrategy.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
.../dolphinscheduler/common/enums/dq/DqTaskState.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...lphinscheduler/common/enums/dq/ExecuteSqlType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...he/dolphinscheduler/common/enums/dq/InputType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...dolphinscheduler/common/enums/dq/OperatorType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...hinscheduler/common/enums/dq/OptionSourceType.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
... and 99 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14d49bc...9db18f8. Read the comment docs.

.github/workflows/ci_ut.yml

…ator-dolphinscheduler into dqs_common

sonarqubecloud · 2021-04-14T06:38:29Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

37.8% Coverage
7.5% Duplication

CalvinKirs

LGTM
It seems that the API is not authenticated. If necessary, it is better to complete it when merged into dev.

* add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA * fix Constants bug * remove the unused class * add unit test * fix code style error * add unit test * refactor data quality common entity * fix code style error * add unit test * close e2e test * fix code smell bug * modify dataquality enum value to 14 in TaskType * add data qualtiy task * update * add getDatasourceOptions interface * fix checkstyle * close e2e test * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * add unit test and alter dataquality task result * fix checkstyle * fix process service test error * add unit test and fix code smells * fix checkstyle error * fix unit test error * fix checkstyle error * change execute sql type name * revert ui pom.xml * fix data quality task error * fix checkstyle error * fix dq task src_connector_type ui select bug * fix spark rw postgresql bug * change mysql driver scope * fix form-create json bug * fix code smell * fix DolphinException Bug * fix ui validate rule and Alert title * fix target connection param bug * fix threshold validate change * add rule input entry index * change statistic_comparison_check logic * remove check type change * add DateExpressionReplaceUtil * fix null point expetion * fix null point expetion * fix test error * add more sql driver * fix test error and remove DateExprReplaceUtil * add get datasource tables and columns * add get datasource tables and columns * remove hive-jdbc in pom.xml * fix code smells * update sql * change the pom.xml * optimize multi_table_accuracy ui * fix v-show error * fix code smells * update sql * [Feature][DataQuality] Add data quality task ui (#5054) * add data quality task ui * update skywalking-eyes SHA * fix style * fix eslint error * fix eslint error * test e2e * fix dq task src_connector_type ui select bug * fix threshold validate change * remove check type change * add get datasource tables and columns * optimize multi_table_accuracy ui * fix v-show error * fix code smells Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality module (#4830) * add data quality module * add license * add package configuration in dist pom * fix license and jar import bug * replace apache/skywalking-eyes@9bd5feb SHA * refacotr jbdc-connector and writer * modify parameter name in HiveConnector * fix checkstyle error * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix checkstyle error in dolphinschesuler-dist * fix duplicate code bug * fix code style bug * fix code smells * update * close e2e test * fix spark rw postgresql bug * change mysql driver scope * add more sql driver * remove hive-jdbc in pom.xml * change the pom.xml Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * [Feature][DataQuality] Add data quality task backend (#4883) * add dq relevant enums and parameter * replace apache/skywalking-eyes@9bd5feb SHA Co-authored-by: sunchaohe <sunzhaohe@linklogis.com> * refactor data_quality_module * add header license * data quality module refactor * fix unit test error * fix checkstyle error * fix unit test error * fix checkstyle error * fix unit test error * fix code smell * fix check style * fix unit test error * task statistics value add unique code * fix unit test error * fix checkstyle error * fix checkstyle * fix security hotspot * fix unit test error * fix security hotspot * fix check * add data quality task error handling * fix unit test error * add unit test * add unit test * optimize data quality result alert * fix unit test * fix sql script error * fix bug * update sql script * fix checkstyle * add license * fix checkstyle * fix checkstyle * fix unit test * add jacoco dependencies * fix unit test * fix unit test * add jacoco dependencies * add unit test * add unit test * add license * fix checkstyle * fix pom * fix checkstyle * fix checkstyle * merge dev * fix ui error * fix pom error * fix pom error * fix test error * fix test error * mssql-jdbc exclude azure-keyvault * fix test error * merge dev and add unit test * add notes * rollback the CollectionUtils * fix * update sql * fix * fix * fix query rule page error * change dq.jar path * fix sql error * fix ui error * fix(dq): jar path&task enum description * add notes on DataQualityApplication * fix dq result jump error * fix(ui): page condition * feat(ui): add show error output path * change version * remove all chinese word in sql * merge Co-authored-by: sunchaohe <sunzhaohe@linklogis.com>

sunchaohe added 5 commits February 25, 2021 17:22

add dq relevant enums and parameter

bf4e1c3

replace apache/skywalking-eyes@9bd5feb SHA

b6286e0

fix Constants bug

e2209cd

remove the unused class

91e991e

add unit test

6928646

zixi0825 changed the title ~~Dqs common~~ [Feature][DataQuality] Add common entity that are used in data quality task Feb 26, 2021

fix code style error

f8e362b

add unit test

419edcc

zixi0825 mentioned this pull request Feb 26, 2021

[Feature] Data Quality Design #4283

Closed

sunchaohe added 6 commits March 3, 2021 19:14

refactor data quality common entity

a319b63

fix code style error

2355e85

add unit test

7fc1a52

close e2e test

118c8b2

fix code smell bug

57aa834

modify dataquality enum value to 14 in TaskType

1f9eec9

CalvinKirs requested changes Mar 9, 2021

View reviewed changes

.github/workflows/ci_ut.yml Outdated Show resolved Hide resolved

sunchaohe added 3 commits March 11, 2021 16:56

add data qualtiy task

8f58f14

add getDatasourceOptions interface

b0c8ca5

fix checkstyle

76f8521

zixi0825 changed the title ~~[Feature][DataQuality] Add common entity that are used in data quality task~~ [Feature][DataQuality] Add data quality task Mar 14, 2021

zixi0825 changed the title ~~[Feature][DataQuality] Add data quality task~~ [Feature][DataQuality] Add data quality task backend Mar 14, 2021

sunchaohe added 8 commits March 17, 2021 18:18

add unit test and alter dataquality task result

ae94f04

fix checkstyle

f73d99d

fix process service test error

a17d57c

add unit test and fix code smells

0cd5795

fix checkstyle error

1c88d08

fix unit test error

0d4896a

fix checkstyle error

00e9a5e

change execute sql type name

ad6ddf4

sunchaohe added 23 commits March 19, 2021 14:48

revert ui pom.xml

8b62434

fix data quality task error

fd8bd1a

fix checkstyle error

7863f3d

fix form-create json bug

e456af0

fix code smell

4ab99e0

fix DolphinException Bug

9db18f8

fix ui validate rule and Alert title

aad137d

fix target connection param bug

7483c9f

add rule input entry index

86a6543

change statistic_comparison_check logic

f861a1c

add DateExpressionReplaceUtil

8d1cee6

fix null point expetion

d960849

fix null point expetion

e6ce977

fix test error

d531569

fix test error and remove DateExprReplaceUtil

87fa888

add get datasource tables and columns

a82cf5f

fix code smells

ee47fbb

update sql

ae868e2

update sql

4769c90

update sql

a70ba5a

update sql

36d940b

Merge branch 'data_quality_design' of https://github.com/apache/incub…

551b4b5

…ator-dolphinscheduler into dqs_common

fix unit test error

5cb2d50

CalvinKirs self-requested a review April 22, 2021 11:20

CalvinKirs approved these changes Apr 27, 2021

View reviewed changes

CalvinKirs merged commit 164d678 into apache:data_quality_design Apr 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][DataQuality] Add data quality task backend#4883

[Feature][DataQuality] Add data quality task backend#4883
CalvinKirs merged 47 commits intoapache:data_quality_designfrom
zixi0825:dqs_common

zixi0825 commented Feb 26, 2021

Uh oh!

codecov-io commented Feb 26, 2021 •

edited

Loading

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 14, 2021

Uh oh!

CalvinKirs left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zixi0825 commented Feb 26, 2021

What is the purpose of the pull request

Uh oh!

codecov-io commented Feb 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 14, 2021

Uh oh!

CalvinKirs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-io commented Feb 26, 2021 •

edited

Loading