Skip to content

[feature](cloud) Support query with freshness tolerance #53540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 35 commits into from
Sep 18, 2025
Merged

[feature](cloud) Support query with freshness tolerance #53540

dataroaring merged 35 commits into from
Sep 18, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jul 18, 2025 •
edited
Loading

What problem does this PR solve?

doc: apache/doris-website#2934

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jul 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 force-pushed the query-freshness-session-var branch 3 times, most recently from 0919575 to 39a0eea Compare July 25, 2025 02:48
dataroaring
dataroaring reviewed Jul 26, 2025
View reviewed changes
dataroaring
dataroaring reviewed Jul 26, 2025
View reviewed changes
@bobhan1 bobhan1 force-pushed the query-freshness-session-var branch 10 times, most recently from 74d1d57 to 0791b6c Compare August 1, 2025 06:52
@bobhan1 bobhan1 force-pushed the query-freshness-session-var branch 5 times, most recently from fa4aa5e to 963a65e Compare August 4, 2025 09:11
gavinchou
gavinchou reviewed Aug 5, 2025
View reviewed changes
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
@zhannngchen zhannngchen mentioned this pull request Aug 13, 2025
Merged
16 tasks
zhannngchen pushed a commit that referenced this pull request Aug 18, 2025
[feat](warm up) introduce immediate warmup on read cluster (#54611)
64b57dc 
### What problem does this PR solve?

Problem Summary:

introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets

NOTE：
the method `get_rowset_warmup_state()` is used for #53540
@bobhan1 bobhan1 force-pushed the query-freshness-session-var branch 2 times, most recently from 69ca66b to 60cc5f6 Compare August 20, 2025 09:23
@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-H: Total hot run time: 1517 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false

------ Round 1 ----------------------------------
q1	17227	29	20	20
q2	2615	26	25	25
q3	10496	16	14	14
q4	11192	16	17	16
q5	8271	15	15	15
q6	236	14	13	13
q7	994	26	24	24
q8	9518	15	13	13
q9	9725	26	13	13
q10	7040	13	12	12
q11	715	21	20	20
q12	694	13	12	12
q13	18678	12	11	11
q14	250	12	11	11
q15	606	13	10	10
q16	1066	992	979	979
q17	565	13	11	11
q18	7686	13	11	11
q19	1697	13	12	12
q20	1940	372	243	243
q21	8956	21	21	21
q22	1093	13	11	11
Total cold run time: 121260 ms
Total hot run time: 1517 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	11	10	10
q2	19	21	20	20
q3	12	11	10	10
q4	13	11	10	10
q5	12	12	11	11
q6	11	11	10	10
q7	19	19	19	19
q8	11	11	11	11
q9	10	11	10	10
q10	10	10	11	10
q11	18	19	18	18
q12	11	11	11	11
q13	10	11	12	11
q14	11	12	12	12
q15	13	11	10	10
q16	1062	1099	1041	1041
q17	12	10	10	10
q18	10	10	10	10
q19	11	10	11	10
q20	1897	1935	1816	1816
q21	22	19	19	19
q22	10	10	10	10
Total cold run time: 3215 ms
Total hot run time: 3099 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-DS: Total hot run time: 2721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false

query1	1085	20	12	12
query2	7128	18	15	15
query3	7628	11	11	11
query4	27326	12	12	12
query5	4514	13	12	12
query6	368	12	10	10
query7	5443	12	10	10
query8	353	18	15	15
query9	9210	11	10	10
query10	735	11	10	10
query11	15929	11	10	10
query12	174	12	10	10
query13	1725	11	9	9
query14	10893	15	12	12
query15	235	10	9	9
query16	7170	11	10	10
query17	1528	12	35	12
query18	2774	10	9	9
query19	204	10	9	9
query20	133	11	9	9
query21	223	13	12	12
query22	4038	10	10	10
query23	33938	20	15	15
query24	9210	12	13	12
query25	722	9	9	9
query26	1065	10	9	9
query27	3405	10	8	8
query28	6295	10	9	9
query29	1196	10	8	8
query30	630	11	9	9
query31	1732	15	12	12
query32	109	11	9	9
query33	1071	12	11	11
query34	1589	819	512	512
query35	901	11	10	10
query36	993	11	10	10
query37	114	36	10	10
query38	3636	9	8	8
query39	1492	719	747	719
query40	227	9	11	9
query41	77	11	10	10
query42	147	10	9	9
query43	499	9	8	8
query44	1348	10	9	9
query45	195	10	9	9
query46	898	10	10	10
query47	1762	10	10	10
query48	410	10	9	9
query49	1132	10	9	9
query50	762	10	10	10
query51	3950	10	9	9
query52	119	10	9	9
query53	252	10	10	10
query54	722	9	10	9
query55	92	9	9	9
query56	324	11	10	10
query57	1174	10	9	9
query58	298	11	9	9
query59	2725	9	9	9
query60	371	10	9	9
query61	174	11	9	9
query62	836	10	9	9
query63	280	10	12	10
query64	4135	10	9	9
query65	4022	9	9	9
query66	1094	11	9	9
query67	16208	12	10	10
query68	3662	11	8	8
query69	727	10	9	9
query70	1399	10	9	9
query71	421	346	342	342
query72	7455	10	9	9
query73	527	9	9	9
query74	9517	11	10	10
query75	3358	10	11	10
query76	2527	10	11	10
query77	1013	11	10	10
query78	9798	11	10	10
query79	1058	15	12	12
query80	764	9	9	9
query81	729	10	9	9
query82	198	10	8	8
query83	290	11	10	10
query84	290	10	8	8
query85	918	9	10	9
query86	359	9	8	8
query87	3879	9	10	9
query88	2901	13	11	11
query89	406	11	11	11
query90	2007	10	9	9
query91	181	9	9	9
query92	76	9	8	8
query93	1070	10	9	9
query94	801	10	9	9
query95	527	10	10	10
query96	433	9	8	8
query97	3156	12	9	9
query98	233	228	223	223
query99	1487	10	11	10
Total cold run time: 283314 ms
Total hot run time: 2721 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

ClickBench: Total hot run time: 0.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false

query1	0.06	0.02	0.01
query2	0.12	0.00	0.01
query3	0.30	0.01	0.01
query4	1.71	0.01	0.00
query5	0.30	0.00	0.00
query6	1.64	0.00	0.01
query7	0.04	0.01	0.00
query8	0.08	0.00	0.00
query9	0.66	0.00	0.01
query10	0.62	0.00	0.00
query11	0.26	0.00	0.01
query12	0.26	0.00	0.01
query13	0.68	0.00	0.00
query14	1.10	0.00	0.00
query15	0.99	0.00	0.00
query16	0.42	0.00	0.00
query17	1.08	0.00	0.00
query18	0.24	0.01	0.00
query19	2.02	0.00	0.00
query20	0.03	0.01	0.00
query21	15.36	0.01	0.00
query22	5.71	0.00	0.00
query23	15.57	0.00	0.00
query24	1.81	0.00	0.00
query25	0.17	0.00	0.00
query26	0.20	0.00	0.01
query27	0.10	0.00	0.00
query28	1.57	0.00	0.00
query29	12.88	0.01	0.00
query30	0.34	0.00	0.01
query31	2.66	0.00	0.00
query32	6.28	0.00	0.00
query33	4.36	0.01	0.01
query34	8.50	0.00	0.01
query35	7.55	0.00	0.00
query36	0.68	0.01	0.00
query37	0.24	0.00	0.00
query38	0.20	0.00	0.00
query39	0.07	0.00	0.00
query40	0.23	0.00	0.00
query41	0.13	0.01	0.01
query42	0.10	0.00	0.00
query43	0.08	0.01	0.00
Total cold run time: 97.4 s
Total hot run time: 0.04 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

BE UT Coverage Report

Increment line coverage 80.08% (390/487) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.34% (17595/33614)
Line Coverage 37.55% (159678/425288)
Region Coverage 32.05% (121484/379020)
Branch Coverage 33.43% (53349/159581)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 72.85% (432/593) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.03% (23334/32853)
Line Coverage 57.37% (243483/424395)
Region Coverage 52.76% (202719/384196)
Branch Coverage 54.42% (87239/160318)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

FE Regression Coverage Report

Increment line coverage 78.46% (51/65) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Sep 18, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by at least one committer and no changes requested.

dataroaring
dataroaring approved these changes Sep 18, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5e98aad into apache:master Sep 18, 2025
29 of 33 checks passed
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feat](warm up) introduce immediate warmup on read cluster (apache#54611
df1efc9 
)

Problem Summary:

introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets

NOTE：
the method `get_rowset_warmup_state()` is used for apache#53540
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
43b9405
@bobhan1 bobhan1 mentioned this pull request Sep 24, 2025
Merged
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
a37b02d
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
7d08978
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Sep 28, 2025
@bobhan1
[feat](warm up) introduce immediate warmup on read cluster (apache#54611
c783468 
)

Problem Summary:

introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets

NOTE：
the method `get_rowset_warmup_state()` is used for apache#53540
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 28, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
62c5649
morrySnow pushed a commit that referenced this pull request Sep 29, 2025
@bobhan1
branch-3.1: [feature](cloud) Support query with freshness tolerance #…
f30eb3d 
…53540 #54611 (#56390)

pick #54611 and #53540, modified due to conflict with #52715

---------

Co-authored-by: zhannngchen <zhangchen@selectdb.com>
@bobhan1 bobhan1 mentioned this pull request Sep 30, 2025
Merged
8 tasks
dataroaring pushed a commit to apache/doris-website that referenced this pull request Sep 30, 2025
@bobhan1
[doc](file-cache) Best Practices for Cache Optimization in Read-Write…
c1dd226 
… Splitting Scenarios (#2934)

modified based on #2745

for apache/doris#53540
## Versions 

- [x] dev
- [ ] 3.0
- [ ] 2.1
- [ ] 2.0

## Languages

- [x] Chinese
- [x] English

## Docs Checklist

- [ ] Checked by AI
- [ ] Test Cases Built

---------

Co-authored-by: Chen Zhang <zhangchen@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

+1 more reviewer

@pingchunzhang pingchunzhang pingchunzhang left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.2-merged dev/4.0.x-conflict reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bobhan1 @Thearas @doris-robot @hello-stephen @dataroaring @gavinchou @pingchunzhang @morrySnow