Skip to content

[GLUTEN-7028][CH][Part-3] Refactor: Move mergetree related codes to backends-clickhouse#7234

Merged
baibaichen merged 13 commits intoapache:mainfrom
baibaichen:feature/one-pipeline-0
Sep 26, 2024
Merged

[GLUTEN-7028][CH][Part-3] Refactor: Move mergetree related codes to backends-clickhouse#7234
baibaichen merged 13 commits intoapache:mainfrom
baibaichen:feature/one-pipeline-0

Conversation

@baibaichen
Copy link
Copy Markdown
Contributor

@baibaichen baibaichen commented Sep 13, 2024

What changes were proposed in this pull request?

This is second refacor PR for moving mergetree related codes to backends-clickhouse:

  1. Move JniUtils from gluten-arrow to gluten-core and using config.proto, so we can use ConfigMap to pass configuration between java and c++.
  2. Move ExtensionTableXXX from gluten-substrait to backends-clickhouse
  3. Move genWriteParameters to TransformerApi, so we can pass mergetree related parameters.

(Fixes: #7028)

How was this patch tested?

Using Existed UTs

@github-actions github-actions bot added CORE works for Gluten Core VELOX CLICKHOUSE labels Sep 13, 2024
@github-actions
Copy link
Copy Markdown

#7028

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from a879269 to 46166be Compare September 18, 2024 07:16
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 46166be to 04682c9 Compare September 18, 2024 13:49
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 04682c9 to 3421f0d Compare September 18, 2024 14:01
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen changed the title [GLUTEN-7028][CH][Part-2] [GLUTEN-7028][CH][Part-3] Sep 19, 2024
@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 79727d7 to 8ce0620 Compare September 19, 2024 11:00
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 8ce0620 to eac0c7b Compare September 19, 2024 11:46
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 4c0e635 to 3961fdc Compare September 19, 2024 14:25
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 3961fdc to 36862c0 Compare September 19, 2024 16:30
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

4 similar comments
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

* limitations under the License.
*/
package org.apache.gluten.init;
package org.apache.gluten.jni;
Copy link
Copy Markdown
Member

@zhztheplayer zhztheplayer Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we move it to gluten-core, could we do some minor refactors on this class at the same time? My suggestion would be:

  1. org.apache.gluten.jni.JniUtils -> org.apache.gluten.utils.ConfigUtil
  2. JniUtils.toNativeConf -> ConfigUtil.serialize or ConfigUtil.serializeMap

Thanks.

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 498934f to 1821993 Compare September 20, 2024 12:04
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 771ec15 to 8b9f159 Compare September 24, 2024 22:41
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 8b9f159 to 6f875ea Compare September 25, 2024 03:32
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from 6f875ea to c50e45a Compare September 25, 2024 06:31
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from c50e45a to ef1fa30 Compare September 25, 2024 08:13
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one-pipeline-0 branch from ef1fa30 to d9678a5 Compare September 25, 2024 11:42
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen changed the title [GLUTEN-7028][CH][Part-3] [GLUTEN-7028][CH][Part-3] Refactor: move mergetree related codes to backends-clickhouse Sep 25, 2024
@baibaichen baibaichen changed the title [GLUTEN-7028][CH][Part-3] Refactor: move mergetree related codes to backends-clickhouse [GLUTEN-7028][CH][Part-3] Refactor: Move mergetree related codes to backends-clickhouse Sep 25, 2024
@baibaichen baibaichen merged commit df24479 into apache:main Sep 26, 2024
@baibaichen baibaichen deleted the feature/one-pipeline-0 branch September 26, 2024 00:59
sharkdtu pushed a commit to sharkdtu/gluten that referenced this pull request Nov 11, 2024
…ackends-clickhouse (apache#7234)

This is second refacor PR for moving mergetree related codes to backends-clickhouse:
1. Move JniUtils from gluten-arrow to gluten-core and using config.proto, so we can use ConfigMap to pass configuration between java and c++.
2. Move ExtensionTableXXX from gluten-substrait to backends-clickhouse
3. Move `genWriteParameters` to `TransformerApi`, so we can pass mergetree related parameters.

(Fixes: \apache#7028)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLICKHOUSE CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CH] Fully Support writing parquet and mergetree in spark 3.5.x with delta protocol

2 participants