You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Function
DataQualityApplication uses spark as execution engine and parse the input parameter to generate Connector、Executor and Writer .Read data from different data source through different Connector and use the sql in executor to operate data. Finally write the result into storage engine through the Writer
Architecture
Parameter design
Parse input parameter to generate ConnectorParamters、ExecutorParameters and WriterParameters
Flow and DataQualityTask
DataQualityTask is a interface, all the connectors、executors and writers must implement the DataQualityTask
A Flow consist of a list of DataQualityTask
The DataQualityApplication is to execute all DataQualityTask in Flow to complete the execution logic