[fix](jdbc catalog) fix a jdbc catalog npe #50901

zy-kkk · 2025-05-14T08:41:52Z

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDbType()" because "this.jdbcClient" is null
    at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.getDatabaseTypeName(JdbcExternalCatalog.java:160) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.jdbc.JdbcExternalCatalog.configureJdbcTable(JdbcExternalCatalog.java:342) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.tablefunction.JdbcQueryTableValueFunction.getScanNode(JdbcQueryTableValueFunction.java:55) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalTVFRelation(PhysicalPlanTranslator.java:942) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalTVFRelation(PhysicalPlanTranslator.java:245) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalTVFRelation.accept(PhysicalTVFRelation.java:117) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:403) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:245) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:72) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:273) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:480) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:569) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:158) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:154) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.qe.StmtExecutor.executeInternalQuery(StmtExecutor.java:3529) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.jdbc.JdbcExternalTable.getRowCount(JdbcExternalTable.java:231) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.jdbc.JdbcExternalTable.fetchRowCount(JdbcExternalTable.java:210) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.ExternalRowCountCache$RowCountCacheLoader.doLoad(ExternalRowCountCache.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.ExternalRowCountCache$RowCountCacheLoader.doLoad(ExternalRowCountCache.java:82) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.statistics.BasicAsyncCacheLoader.lambda$asyncLoad$0(BasicAsyncCacheLoader.java:39) ~[doris-fe.jar:1.2-SNAPSHOT]
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]
2025-04-29 07:30:25,586 WARN (NotCheckpointRowCountRefreshExecutor-48|190) [JdbcExternalTable.getRowCount():257] Failed to fetch row count for table test_lower_case_mtmv.external_lower_mtmv.table_test. Reason [Failed to execute internal SQL. java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getDbType()" because "this.jdbcClient" is null]
2025-04-29 07:30:25,587 WARN (mtmv-task-execute-1-thread-2|190) [CollectRelation.collectFromUnboundRelation():190] collect insert target table '[internal, EXTERNAL_LOWER_MTMV, MTMV_TEST]' more than once.
2025-04-29 07:30:25,588 WARN (mtmv-task-execute-1-thread-2|190) [InsertOverwriteTableCommand.run():241] insert into overwrite failed with task(or group) id 1745880283332
2025-04-29 07:30:25,603 ERROR (mtmv-task-execute-1-thread-2|190) [MTMVTask.run():234] Execution failed after retries: Cannot invoke "org.apache.doris.datasource.jdbc.client.JdbcClient.getColumnsFromJdbc(String, String)" because "this.jdbcClient" is null

Thearas · 2025-05-14T08:42:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zy-kkk · 2025-05-14T08:42:13Z

run buildall

morningman

LGTM

github-actions · 2025-05-14T08:52:20Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-14T08:52:22Z

PR approved by anyone and no changes requested.

doris-robot · 2025-05-14T09:01:29Z

TPC-H: Total hot run time: 33544 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9200e96670afd4d8805636f77abacb4e2fb1b2a0, data reload: false

------ Round 1 ----------------------------------
q1	26109	4996	4924	4924
q2	2067	271	182	182
q3	10398	1207	696	696
q4	10231	974	515	515
q5	7541	2346	2320	2320
q6	183	169	132	132
q7	918	742	602	602
q8	9317	1304	1084	1084
q9	6711	5100	5148	5100
q10	6856	2291	1873	1873
q11	486	279	264	264
q12	343	343	206	206
q13	17788	3633	3106	3106
q14	234	241	213	213
q15	526	485	486	485
q16	417	441	376	376
q17	582	848	347	347
q18	7660	7245	7013	7013
q19	1749	949	554	554
q20	331	343	219	219
q21	3836	2569	2376	2376
q22	1064	1011	957	957
Total cold run time: 115347 ms
Total hot run time: 33544 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5095	5038	5066	5038
q2	232	331	231	231
q3	2155	2649	2270	2270
q4	1353	1750	1336	1336
q5	4381	4500	4434	4434
q6	212	169	129	129
q7	1956	1906	1717	1717
q8	2564	2486	2445	2445
q9	7166	7028	7184	7028
q10	3012	3192	2747	2747
q11	555	498	498	498
q12	672	760	612	612
q13	3425	3843	3350	3350
q14	277	296	271	271
q15	521	490	472	472
q16	430	474	430	430
q17	1120	1544	1360	1360
q18	7620	7432	7324	7324
q19	796	779	892	779
q20	2001	2054	1889	1889
q21	5130	4631	4606	4606
q22	1068	1037	1001	1001
Total cold run time: 51741 ms
Total hot run time: 49967 ms

doris-robot · 2025-05-14T09:13:01Z

TPC-DS: Total hot run time: 186209 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9200e96670afd4d8805636f77abacb4e2fb1b2a0, data reload: false

query1	1013	488	504	488
query2	6569	1829	1814	1814
query3	6737	216	213	213
query4	26462	23301	22977	22977
query5	4321	622	454	454
query6	307	214	200	200
query7	4617	484	277	277
query8	293	232	249	232
query9	8588	2632	2673	2632
query10	463	311	263	263
query11	15719	15094	14976	14976
query12	162	110	103	103
query13	1640	528	402	402
query14	8927	6081	6002	6002
query15	195	186	173	173
query16	7136	632	478	478
query17	1193	714	567	567
query18	1965	429	294	294
query19	184	184	165	165
query20	118	120	117	117
query21	218	126	111	111
query22	4103	4236	4042	4042
query23	33941	32814	32931	32814
query24	7695	2375	2379	2375
query25	541	449	386	386
query26	1232	260	148	148
query27	2704	484	338	338
query28	4316	2130	2116	2116
query29	770	550	467	467
query30	282	217	197	197
query31	922	865	752	752
query32	74	62	64	62
query33	555	366	297	297
query34	803	823	525	525
query35	764	798	732	732
query36	935	983	888	888
query37	118	101	74	74
query38	4081	4192	4138	4138
query39	1455	1421	1430	1421
query40	206	122	106	106
query41	80	53	52	52
query42	127	114	109	109
query43	483	510	480	480
query44	1287	821	823	821
query45	176	180	171	171
query46	846	1027	647	647
query47	1735	1809	1704	1704
query48	385	414	322	322
query49	784	521	437	437
query50	628	667	408	408
query51	4131	4211	4074	4074
query52	106	105	97	97
query53	216	244	183	183
query54	598	574	524	524
query55	88	85	89	85
query56	343	327	299	299
query57	1139	1140	1143	1140
query58	296	256	250	250
query59	2685	2605	2492	2492
query60	330	304	313	304
query61	124	119	126	119
query62	790	724	665	665
query63	228	187	189	187
query64	4340	1006	673	673
query65	4328	4267	4275	4267
query66	1143	421	327	327
query67	15745	15550	15540	15540
query68	6029	898	534	534
query69	493	327	270	270
query70	1147	1143	1166	1143
query71	407	324	300	300
query72	5948	4833	4650	4650
query73	615	564	356	356
query74	8818	9019	8902	8902
query75	3183	3257	2710	2710
query76	3195	1182	747	747
query77	478	372	298	298
query78	10040	10235	9358	9358
query79	2566	831	594	594
query80	1603	537	461	461
query81	562	255	216	216
query82	390	131	98	98
query83	351	264	234	234
query84	246	106	86	86
query85	897	363	329	329
query86	461	285	285	285
query87	4430	4571	4367	4367
query88	3439	2346	2302	2302
query89	390	308	287	287
query90	1932	216	219	216
query91	145	144	113	113
query92	78	64	57	57
query93	2626	936	585	585
query94	750	420	319	319
query95	382	297	282	282
query96	498	580	289	289
query97	3108	3221	3155	3155
query98	246	209	220	209
query99	1341	1435	1264	1264
Total cold run time: 272279 ms
Total hot run time: 186209 ms

doris-robot · 2025-05-14T09:18:13Z

ClickBench: Total hot run time: 29.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9200e96670afd4d8805636f77abacb4e2fb1b2a0, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.11
query3	0.26	0.20	0.19
query4	1.59	0.19	0.12
query5	0.55	0.56	0.55
query6	1.21	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.50
query10	0.56	0.56	0.55
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.80	0.83	0.80
query15	0.89	0.86	0.86
query16	0.41	0.41	0.38
query17	1.04	1.02	1.05
query18	0.23	0.21	0.21
query19	1.94	1.80	1.87
query20	0.01	0.01	0.01
query21	15.41	0.92	0.58
query22	0.77	1.25	0.71
query23	14.78	1.36	0.64
query24	7.37	1.31	0.85
query25	0.44	0.19	0.10
query26	0.62	0.16	0.14
query27	0.06	0.06	0.05
query28	9.09	0.88	0.44
query29	12.62	4.00	3.29
query30	0.26	0.10	0.07
query31	2.82	0.61	0.39
query32	3.23	0.55	0.46
query33	3.26	3.17	3.09
query34	15.66	5.15	4.47
query35	4.58	4.51	4.47
query36	0.65	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.33 s
Total hot run time: 29.26 s

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

cherry-pick #50901

Cherry-picked from #50901 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

### What problem does this PR solve? Related PR: #50901 Problem Summary: In the previous PR, I added makeSureInitialized() to configureJdbcTable, which will create the jdbcclient if it does not exist. However, in the testJdbcConnection process, makeSureInitialized() is bypassed and initLocalObjectsImpl is used directly. This will cause a jdbcclient to be created again when configureJdbcTable is used in testJdbcConnection, and the previous jdbcclient is not closed, resulting in a connection leak. In the modification of this PR, I used a completely independent testClient for the TestConnection process to avoid this problem.

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

…#51036) ### What problem does this PR solve? Related PR: apache#50901 Problem Summary: In the previous PR, I added makeSureInitialized() to configureJdbcTable, which will create the jdbcclient if it does not exist. However, in the testJdbcConnection process, makeSureInitialized() is bypassed and initLocalObjectsImpl is used directly. This will cause a jdbcclient to be created again when configureJdbcTable is used in testJdbcConnection, and the previous jdbcclient is not closed, resulting in a connection leak. In the modification of this PR, I used a completely independent testClient for the TestConnection process to avoid this problem.

[fix](jdbc catalog) fix a jdbc catalog npe

9200e96

morningman approved these changes May 14, 2025

View reviewed changes

morningman added dev/2.1.x dev/3.0.x labels May 14, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2025

github-actions bot added the reviewed label May 14, 2025

This was referenced May 14, 2025

branch-3.0: [fix](jdbc catalog) fix a jdbc catalog npe #50911

Merged

branch-2.1: [fix](jdbc catalog) fix a jdbc catalog npe #50912

Closed

hubgeter approved these changes May 15, 2025

View reviewed changes

wuwenchi approved these changes May 15, 2025

View reviewed changes

morningman merged commit a2f79fe into apache:master May 15, 2025
32 of 33 checks passed

morningman added the p0_c label May 15, 2025

github-actions bot pushed a commit that referenced this pull request May 15, 2025

[fix](jdbc catalog) fix a jdbc catalog npe (#50901)

7a60583

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

github-actions bot mentioned this pull request May 15, 2025

branch-3.0: [fix](jdbc catalog) fix a jdbc catalog npe #50901 #50927

Closed

github-actions bot pushed a commit that referenced this pull request May 15, 2025

[fix](jdbc catalog) fix a jdbc catalog npe (#50901)

31e18b4

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

github-actions bot mentioned this pull request May 15, 2025

branch-2.1: [fix](jdbc catalog) fix a jdbc catalog npe #50901 #50928

Merged

dataroaring pushed a commit that referenced this pull request May 15, 2025

branch-3.0: [fix](jdbc catalog) fix a jdbc catalog npe (#50911)

9f09730

cherry-pick #50901

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 15, 2025

yiguolei pushed a commit that referenced this pull request May 17, 2025

branch-2.1: [fix](jdbc catalog) fix a jdbc catalog npe #50901 (#50928)

e3fbf39

Cherry-picked from #50901 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

yiguolei added dev/2.1.11-merged and removed dev/2.1.x labels May 17, 2025

zy-kkk deleted the fix_npe_0514 branch May 19, 2025 09:34

This was referenced May 19, 2025

[hotfix](jdbc catalog) Fix jdbcclient repeated initialization #51036

Merged

branch-3.0: [hotfix](jdbc catalog) Fix jdbcclient repeated initialization #51037

Merged

branch-2.1: [hotfix](jdbc catalog) Fix jdbcclient repeated initialization #51038

Merged

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](jdbc catalog) fix a jdbc catalog npe (apache#50901)

0c77139

We must add makeSureInitialized() in the fetchRowCount() method to ensure the initialization of catalog related resources.

yiguolei mentioned this pull request Aug 11, 2025

2.1.11 Release Notes #54550

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix](jdbc catalog) fix a jdbc catalog npe #50901

[fix](jdbc catalog) fix a jdbc catalog npe #50901

Uh oh!

zy-kkk commented May 14, 2025 •

edited

Loading

Uh oh!

Thearas commented May 14, 2025

Uh oh!

zy-kkk commented May 14, 2025

Uh oh!

morningman left a comment

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[fix](jdbc catalog) fix a jdbc catalog npe #50901

[fix](jdbc catalog) fix a jdbc catalog npe #50901

Uh oh!

Conversation

zy-kkk commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented May 14, 2025

Uh oh!

zy-kkk commented May 14, 2025

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

zy-kkk commented May 14, 2025 •

edited

Loading