Skip to content

branch-3.1: [fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties #56639 #57399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 30, 2025
Merged

branch-3.1: [fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties #56639 #57399

morrySnow merged 1 commit into from
Oct 30, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 28, 2025

Cherry-picked from #56639

@zy-kkk
[fix](external catalog) Distinguish between functions used to refresh…
11dadbe 
… cache and change catalog properties (#56639)

The `resetToUninitialized` method should only be called when altering catalog properties, as it resets the entire catalog to an uninitialized state. Previously, `resetToUninitialized` was mistakenly called in `refreshMgr`. Refresh should only refresh the cache, not the entire catalog. Resetting the entire catalog could cause certain clients to shut down during a cache refresh, leading to errors.

This PR mainly changes:
1. Change the `onRefreshCache()` method
    This method will invalid db/table names cache and all meta cache
2. For `refresh` and `hms listener`, only call `onRefreshCache()`
3. For `alter catalog`, call `resetToUninitialized()`

TODO:
There is still a problem when we `alter catalog`, the ongoing task may be failed or stucked.
@github-actions github-actions bot requested a review from morrySnow as a code owner October 28, 2025 03:52
@Thearas
Copy link
Contributor

Thearas commented Oct 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 28, 2025
@Thearas
Copy link
Contributor

Thearas commented Oct 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 28, 2025

TPC-H: Total hot run time: 34668 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11dadbef4c023122486bc2d74b245f3f6618d00a, data reload: false

------ Round 1 ----------------------------------
q1	17630	5702	5602	5602
q2	2049	439	319	319
q3	12003	1310	817	817
q4	10566	908	532	532
q5	9518	2504	2277	2277
q6	208	177	144	144
q7	993	804	685	685
q8	9393	1627	1295	1295
q9	5487	5128	4977	4977
q10	6860	2296	1877	1877
q11	538	325	306	306
q12	393	406	248	248
q13	17806	3704	3172	3172
q14	250	243	237	237
q15	533	490	469	469
q16	472	484	422	422
q17	744	932	449	449
q18	7290	6573	6690	6573
q19	1228	1008	635	635
q20	409	383	241	241
q21	3781	2500	2347	2347
q22	1134	1088	1044	1044
Total cold run time: 109285 ms
Total hot run time: 34668 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5679	5616	5559	5559
q2	251	342	253	253
q3	2346	2711	2453	2453
q4	1449	1881	1546	1546
q5	4839	5207	5126	5126
q6	191	172	136	136
q7	2121	2011	1866	1866
q8	2778	2905	2816	2816
q9	7433	7352	7350	7350
q10	3051	3371	2854	2854
q11	631	551	533	533
q12	718	797	643	643
q13	3570	3910	3313	3313
q14	321	316	281	281
q15	548	490	479	479
q16	508	530	470	470
q17	1332	1779	1323	1323
q18	7859	7762	7568	7568
q19	986	1272	1190	1190
q20	2163	2112	1967	1967
q21	6232	5243	4934	4934
q22	1227	1128	1121	1121
Total cold run time: 56233 ms
Total hot run time: 53781 ms

@doris-robot
Copy link

doris-robot commented Oct 28, 2025

ClickBench: Total hot run time: 31.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11dadbef4c023122486bc2d74b245f3f6618d00a, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.07
query4	1.60	0.12	0.11
query5	0.54	0.53	0.55
query6	1.13	0.75	0.73
query7	0.03	0.02	0.02
query8	0.06	0.03	0.04
query9	0.64	0.54	0.55
query10	0.62	0.59	0.59
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.64	0.62	0.62
query14	0.81	0.84	0.81
query15	0.88	0.87	0.87
query16	0.42	0.41	0.41
query17	1.14	1.14	1.12
query18	0.27	0.25	0.23
query19	2.08	1.96	2.12
query20	0.01	0.01	0.01
query21	15.36	1.09	0.68
query22	0.74	0.86	0.84
query23	14.85	1.56	0.62
query24	2.73	1.21	1.36
query25	0.15	0.07	0.07
query26	0.43	0.18	0.16
query27	0.05	0.05	0.06
query28	13.09	1.18	0.48
query29	12.55	4.67	3.81
query30	0.26	0.10	0.08
query31	2.85	0.68	0.41
query32	3.23	0.59	0.50
query33	3.06	3.15	3.14
query34	16.67	5.22	4.55
query35	4.63	4.62	4.66
query36	0.68	0.54	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 103.4 s
Total hot run time: 31.01 s

@morrySnow morrySnow merged commit 4d20bc8 into branch-3.1 Oct 30, 2025
23 checks passed
@morrySnow morrySnow deleted the auto-pick-56639-branch-3.1 branch October 30, 2025 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @morrySnow @dataroaring @zy-kkk