Skip to content

[GLUTEN-6067][CH][Part 1] Support CH backend with Spark3.5#6068

Merged
zzcclp merged 2 commits intoapache:mainfrom
zzcclp:support_spark35_delta32
Jun 14, 2024
Merged

[GLUTEN-6067][CH][Part 1] Support CH backend with Spark3.5#6068
zzcclp merged 2 commits intoapache:mainfrom
zzcclp:support_spark35_delta32

Conversation

@zzcclp
Copy link
Copy Markdown
Contributor

@zzcclp zzcclp commented Jun 13, 2024

What changes were proposed in this pull request?

Support CH backend with Spark3.5:

  1. Upgrade Spark version to 3.5 and compile passed;
  2. Upgrade Delta version to 3.2 and compile passed;

How was this patch tested?

for task 1 and task 2, manual tests

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@github-actions
Copy link
Copy Markdown

#6067

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from bf599c0 to 85d3483 Compare June 13, 2024 05:37
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 85d3483 to 76c1227 Compare June 13, 2024 09:57
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 76c1227 to 86269ac Compare June 13, 2024 12:03
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

zzcclp added 2 commits June 14, 2024 09:45
Support CH backend with Spark3.5:

1. Upgrade Spark version to 3.5 and compile passed;
2. Upgrade Delta version to 3.2 and compile passed;
3. CH backend UT passed: (now only the MergeTree + Delta UT passed);
4. Parquet native write passed;
5. Gluten UT passed;
6. Support to run Gluten CH CI with Spark 3.5
@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 6871923 to ca9581b Compare June 14, 2024 01:57
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@baibaichen baibaichen changed the title [GLUTEN-6067][CH] Support CH backend with Spark3.5 (Task 1 and Taks 2) [GLUTEN-6067][CH][Part 1] Support CH backend with Spark3.5 Jun 14, 2024
Copy link
Copy Markdown
Contributor

@baibaichen baibaichen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zzcclp zzcclp closed this Jun 14, 2024
@zzcclp zzcclp reopened this Jun 14, 2024
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@zzcclp
Copy link
Copy Markdown
Contributor Author

zzcclp commented Jun 14, 2024

misoperation, merging this pr, all ci already passed

@zzcclp zzcclp merged commit e52bda6 into apache:main Jun 14, 2024
@GlutenPerfBot
Copy link
Copy Markdown
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_6068_time.csv log/native_master_06_13_2024_142cf0fbc_time.csv difference percentage
q1 34.22 35.57 1.350 103.95%
q2 25.37 23.78 -1.591 93.73%
q3 37.15 39.61 2.453 106.60%
q4 34.36 32.76 -1.599 95.35%
q5 68.71 67.54 -1.167 98.30%
q6 8.47 9.42 0.946 111.17%
q7 84.00 80.69 -3.302 96.07%
q8 86.52 84.77 -1.751 97.98%
q9 120.20 120.70 0.503 100.42%
q10 45.40 45.25 -0.149 99.67%
q11 20.45 19.79 -0.667 96.74%
q12 26.52 27.92 1.393 105.25%
q13 39.84 39.43 -0.407 98.98%
q14 18.95 18.98 0.039 100.20%
q15 32.82 33.04 0.227 100.69%
q16 13.41 13.85 0.440 103.28%
q17 104.75 101.57 -3.183 96.96%
q18 145.85 144.97 -0.878 99.40%
q19 15.18 13.79 -1.386 90.87%
q20 27.99 26.68 -1.309 95.32%
q21 261.57 260.83 -0.747 99.71%
q22 14.21 13.91 -0.299 97.89%
total 1265.94 1254.85 -11.086 99.12%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants