[fix](be java) be hang while enable_java_support #52818

cambyzju · 2025-07-05T01:23:39Z

What problem does this PR solve?

Problem

after enable java support, be can not start correctly, it will hang on stack:

(gdb) bt
#0  0x00007f5fb1e97ce6 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f5fb1e9a798 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007f5fb2c98bf3 in os::PlatformEvent::park() () from /usr/lib/jvm/java-17//lib/server/libjvm.so
#3  0x00007f5fb2c693a5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so
#4  0x00007f5fb2e8b316 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so
#5  0x00007f5fb2934a97 in JVM_MonitorWait () from /usr/lib/jvm/java-17//lib/server/libjvm.so
#6  0x00007f5f9de245ba in ?? ()
#7  0x00007f5f49446158 in ?? ()
#8  0x00007f5faeb9fa00 in ?? ()
#9  0x00007ffc37f91178 in ?? ()
#10 0x00007f5f9de304bd in ?? ()
#11 0x00007ffc37f910a0 in ?? ()
#12 0x0000000000000000 in ?? ()

jstack of be:

"main" #1 prio=5 os_prio=0 cpu=931.38ms elapsed=66.08s tid=0x00007fab8e12c400 nid=0x3e68aa in Object.wait()  [0x00007ffdb3d4c000]  
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.15-ga/Native Method)
    - waiting on <0x00000000bd183b68> (a java.lang.ProcessImpl)
    at java.lang.Object.wait(java.base@17.0.15-ga/Object.java:338)
    at java.lang.ProcessImpl.waitFor(java.base@17.0.15-ga/ProcessImpl.java:434)
    - locked <0x00000000bd183b68> (a java.lang.ProcessImpl)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:1061)
    at org.apache.hadoop.util.Shell.run(Shell.java:957)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282)
    at org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:853)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:838)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
    at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1713)
    at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:103)
    at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:92)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:312)
    - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300)
    - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575)
    at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:279)
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@17.0.15-ga/Native Method)
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/NativeConstructorAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@17.0.15-ga/Constructor.java:500)
    at java.lang.reflect.Constructor.newInstance(java.base@17.0.15-ga/Constructor.java:481)
    at java.util.ServiceLoader$ProviderImpl.newInstance(java.base@17.0.15-ga/ServiceLoader.java:789)
    at java.util.ServiceLoader$ProviderImpl.get(java.base@17.0.15-ga/ServiceLoader.java:729)
    at java.util.ServiceLoader$3.next(java.base@17.0.15-ga/ServiceLoader.java:1403)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3534)
    - locked <0x00000000826529f0> (a java.lang.Class for org.apache.hadoop.fs.FileSystem)

This problem comes from: #45287

after this fix, we could enable java support: #52412

Another Fix Method

Add JAVA_OPTS -Djdk.lang.processReaperUseDefaultStackSize=true inside be.conf, also can fix this problem:

https://bugs.openjdk.org/browse/JDK-8153057

From gemini:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-05T01:23:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

cambyzju · 2025-07-05T01:24:53Z

run buildall

doris-robot · 2025-07-05T02:00:51Z

TPC-H: Total hot run time: 33610 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3907e0d2197b4147e7f006e3b58a450a74796115, data reload: false

------ Round 1 ----------------------------------
q1	17603	5286	5045	5045
q2	1925	274	177	177
q3	10335	1317	704	704
q4	10236	1030	521	521
q5	7979	2386	2333	2333
q6	187	162	131	131
q7	877	742	594	594
q8	9290	1197	1058	1058
q9	6726	5069	5158	5069
q10	6887	2384	1952	1952
q11	483	297	274	274
q12	346	352	213	213
q13	17765	3689	3071	3071
q14	239	223	221	221
q15	561	477	472	472
q16	417	440	387	387
q17	609	845	383	383
q18	7478	7073	6981	6981
q19	1582	957	534	534
q20	335	344	221	221
q21	3841	2534	2308	2308
q22	1057	1047	961	961
Total cold run time: 106758 ms
Total hot run time: 33610 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5208	5151	5147	5147
q2	241	330	219	219
q3	2167	2659	2292	2292
q4	1375	1769	1320	1320
q5	4183	4454	4560	4454
q6	218	172	128	128
q7	1954	2059	1860	1860
q8	2640	2451	2552	2451
q9	7226	7317	7436	7317
q10	3243	3367	2946	2946
q11	584	563	503	503
q12	768	821	663	663
q13	3740	4053	3400	3400
q14	285	305	275	275
q15	538	487	465	465
q16	458	493	436	436
q17	1181	1552	1423	1423
q18	7874	7675	7127	7127
q19	755	740	746	740
q20	1905	1975	1805	1805
q21	4740	4299	4365	4299
q22	1073	1038	991	991
Total cold run time: 52356 ms
Total hot run time: 50261 ms

doris-robot · 2025-07-05T02:12:27Z

TPC-DS: Total hot run time: 186303 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3907e0d2197b4147e7f006e3b58a450a74796115, data reload: false

query1	1027	408	418	408
query2	6523	1639	1680	1639
query3	6737	217	208	208
query4	26776	24279	23247	23247
query5	4307	599	458	458
query6	301	211	203	203
query7	4619	535	291	291
query8	288	229	214	214
query9	8612	2630	2638	2630
query10	445	344	283	283
query11	15781	15258	14900	14900
query12	160	113	108	108
query13	1675	553	424	424
query14	9154	6035	6144	6035
query15	203	193	178	178
query16	7623	654	469	469
query17	1162	735	584	584
query18	1996	418	302	302
query19	200	189	155	155
query20	124	127	133	127
query21	211	140	118	118
query22	3990	4034	4084	4034
query23	34187	33126	32972	32972
query24	8596	2447	2410	2410
query25	551	493	408	408
query26	1239	288	148	148
query27	2752	540	347	347
query28	4272	2130	2102	2102
query29	743	578	457	457
query30	289	224	201	201
query31	912	845	756	756
query32	71	64	65	64
query33	567	407	312	312
query34	807	901	530	530
query35	823	834	742	742
query36	987	1062	919	919
query37	115	105	80	80
query38	4164	4117	4154	4117
query39	1657	1498	1462	1462
query40	210	119	109	109
query41	58	60	59	59
query42	138	119	110	110
query43	492	521	481	481
query44	1380	835	833	833
query45	186	176	169	169
query46	893	1102	665	665
query47	1791	1823	1719	1719
query48	374	426	312	312
query49	770	523	418	418
query50	663	727	423	423
query51	4214	4349	4252	4252
query52	109	109	94	94
query53	233	261	188	188
query54	589	567	496	496
query55	84	83	81	81
query56	317	302	298	298
query57	1191	1228	1146	1146
query58	274	256	262	256
query59	2637	2691	2601	2601
query60	335	333	309	309
query61	134	147	162	147
query62	878	773	723	723
query63	239	190	196	190
query64	4240	990	660	660
query65	4404	4268	4238	4238
query66	1085	444	349	349
query67	15576	15479	15256	15256
query68	7879	894	529	529
query69	490	311	274	274
query70	1181	1115	1128	1115
query71	401	328	298	298
query72	5600	4720	4643	4643
query73	633	586	356	356
query74	9172	8997	8919	8919
query75	3383	3365	2814	2814
query76	3188	1289	788	788
query77	503	393	305	305
query78	10114	10460	9403	9403
query79	1476	873	604	604
query80	610	585	453	453
query81	519	262	232	232
query82	208	131	98	98
query83	259	259	237	237
query84	244	104	88	88
query85	762	363	322	322
query86	373	324	309	309
query87	4356	4531	4267	4267
query88	2960	2315	2299	2299
query89	421	350	305	305
query90	1977	217	211	211
query91	140	144	112	112
query92	74	64	56	56
query93	1719	970	595	595
query94	678	411	307	307
query95	382	294	283	283
query96	494	623	281	281
query97	2766	2784	2629	2629
query98	225	214	202	202
query99	1421	1465	1345	1345
Total cold run time: 272748 ms
Total hot run time: 186303 ms

doris-robot · 2025-07-05T02:17:44Z

ClickBench: Total hot run time: 29.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3907e0d2197b4147e7f006e3b58a450a74796115, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.05	0.04
query3	0.24	0.08	0.08
query4	1.62	0.11	0.11
query5	0.43	0.44	0.42
query6	1.16	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.51	0.53
query10	0.58	0.58	0.58
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.61	0.61
query14	0.80	0.81	0.83
query15	0.95	0.88	0.87
query16	0.38	0.39	0.39
query17	1.10	1.09	1.05
query18	0.24	0.22	0.22
query19	1.91	1.85	1.87
query20	0.02	0.01	0.02
query21	15.40	0.93	0.54
query22	0.75	1.20	0.66
query23	14.95	1.38	0.67
query24	6.64	2.01	0.72
query25	0.51	0.26	0.06
query26	0.67	0.17	0.14
query27	0.06	0.05	0.05
query28	9.17	0.90	0.45
query29	12.52	4.02	3.36
query30	0.25	0.09	0.08
query31	2.83	0.62	0.39
query32	3.23	0.55	0.46
query33	3.06	3.12	3.12
query34	16.08	5.43	4.82
query35	4.87	4.89	4.89
query36	0.70	0.50	0.50
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.32 s
Total hot run time: 29.75 s

cambyzju · 2025-07-05T02:55:48Z

run p0

hello-stephen · 2025-07-05T03:25:41Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.34% (15484/27006)
Line Coverage	46.30% (140654/303771)
Region Coverage	45.59% (71176/156112)
Branch Coverage	40.30% (37496/93036)

github-actions · 2025-07-05T08:01:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-05T08:01:05Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? ### Problem after enable java support, be can not start correctly, it will hang on stack: ``` (gdb) bt #0 0x00007f5fb1e97ce6 in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f5fb1e9a798 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f5fb2c98bf3 in os::PlatformEvent::park() () from /usr/lib/jvm/java-17//lib/server/libjvm.so #3 0x00007f5fb2c693a5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so #4 0x00007f5fb2e8b316 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so #5 0x00007f5fb2934a97 in JVM_MonitorWait () from /usr/lib/jvm/java-17//lib/server/libjvm.so #6 0x00007f5f9de245ba in ?? () #7 0x00007f5f49446158 in ?? () #8 0x00007f5faeb9fa00 in ?? () #9 0x00007ffc37f91178 in ?? () #10 0x00007f5f9de304bd in ?? () #11 0x00007ffc37f910a0 in ?? () #12 0x0000000000000000 in ?? () ``` jstack of be: ``` "main" #1 prio=5 os_prio=0 cpu=931.38ms elapsed=66.08s tid=0x00007fab8e12c400 nid=0x3e68aa in Object.wait() [0x00007ffdb3d4c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@17.0.15-ga/Native Method) - waiting on <0x00000000bd183b68> (a java.lang.ProcessImpl) at java.lang.Object.wait(java.base@17.0.15-ga/Object.java:338) at java.lang.ProcessImpl.waitFor(java.base@17.0.15-ga/ProcessImpl.java:434) - locked <0x00000000bd183b68> (a java.lang.ProcessImpl) at org.apache.hadoop.util.Shell.runCommand(Shell.java:1061) at org.apache.hadoop.util.Shell.run(Shell.java:957) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282) at org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:853) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:838) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1713) at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:103) at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:92) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:312) - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300) - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575) at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:279) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@17.0.15-ga/Native Method) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/NativeConstructorAccessorImpl.java:77) at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@17.0.15-ga/DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@17.0.15-ga/Constructor.java:500) at java.lang.reflect.Constructor.newInstance(java.base@17.0.15-ga/Constructor.java:481) at java.util.ServiceLoader$ProviderImpl.newInstance(java.base@17.0.15-ga/ServiceLoader.java:789) at java.util.ServiceLoader$ProviderImpl.get(java.base@17.0.15-ga/ServiceLoader.java:729) at java.util.ServiceLoader$3.next(java.base@17.0.15-ga/ServiceLoader.java:1403) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3534) - locked <0x00000000826529f0> (a java.lang.Class for org.apache.hadoop.fs.FileSystem) ``` This problem comes from: #45287 after this fix, we could enable java support: #52412 ### Another Fix Method Add JAVA_OPTS `-Djdk.lang.processReaperUseDefaultStackSize=true` inside be.conf, also can fix this problem: https://bugs.openjdk.org/browse/JDK-8153057 From gemini: ![Clipboard_Screenshot_1751683706](https://github.com/user-attachments/assets/a54ffb3a-cb57-4f8a-bc85-5366081c2d9b)

…52823) Cherry-picked from #52818 Co-authored-by: camby <104178625@qq.com>

…52824) Cherry-picked from #52818 Co-authored-by: camby <104178625@qq.com>

fix be hang while enable_java_support

3907e0d

cambyzju added dev/3.0.x dev/3.1.x labels Jul 5, 2025

yiguolei approved these changes Jul 5, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2025

github-actions bot added the reviewed label Jul 5, 2025

zclllyybb approved these changes Jul 5, 2025

View reviewed changes

yiguolei merged commit e98e05d into apache:master Jul 5, 2025
31 of 34 checks passed

github-actions bot mentioned this pull request Jul 5, 2025

branch-3.0: [fix](be java) be hang while enable_java_support #52818 #52823

Merged

github-actions bot mentioned this pull request Jul 5, 2025

branch-3.1: [fix](be java) be hang while enable_java_support #52818 #52824

Merged

dataroaring pushed a commit that referenced this pull request Jul 7, 2025

branch-3.0: [fix](be java) be hang while enable_java_support #52818 (#…

774808f

…52823) Cherry-picked from #52818 Co-authored-by: camby <104178625@qq.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jul 7, 2025

morrySnow pushed a commit that referenced this pull request Jul 8, 2025

branch-3.1: [fix](be java) be hang while enable_java_support #52818 (#…

300105b

…52824) Cherry-picked from #52818 Co-authored-by: camby <104178625@qq.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix](be java) be hang while enable_java_support #52818

[fix](be java) be hang while enable_java_support #52818

Uh oh!

cambyzju commented Jul 5, 2025 •

edited

Loading

Uh oh!

Thearas commented Jul 5, 2025

Uh oh!

cambyzju commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

cambyzju commented Jul 5, 2025

Uh oh!

hello-stephen commented Jul 5, 2025

Uh oh!

github-actions bot commented Jul 5, 2025

Uh oh!

github-actions bot commented Jul 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[fix](be java) be hang while enable_java_support #52818

[fix](be java) be hang while enable_java_support #52818

Uh oh!

Conversation

cambyzju commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Problem

Another Fix Method

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 5, 2025

Uh oh!

cambyzju commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

doris-robot commented Jul 5, 2025

Uh oh!

cambyzju commented Jul 5, 2025

Uh oh!

hello-stephen commented Jul 5, 2025

BE UT Coverage Report

Uh oh!

github-actions bot commented Jul 5, 2025

Uh oh!

github-actions bot commented Jul 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

cambyzju commented Jul 5, 2025 •

edited

Loading