Skip to content

branch-3.1: [Enhancement](json load) Set jsonload's default behavior to be read_json_by_line #55861 #56736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 11, 2025
Merged

branch-3.1: [Enhancement](json load) Set jsonload's default behavior to be read_json_by_line #55861 #56736

morrySnow merged 1 commit into from
Oct 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 9, 2025

Cherry-picked from #55861

@0AyanamiRei
[Enhancement](json load) Set jsonload's default behavior to be read_j…
d011538 
…son_by_line (#55861)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR:
[doc-2874](apache/doris-website#2874)

Problem Summary:

For the read_json_by_line and strip_outer_array parameters, considering
that the first parameter will gradually be deprecated in the future, and
some users may forget to specify these two parameters when importing
JSON files, I will modify the default behavior of these two parameters:
if the user does not specify values for these two parameters, the
default setting for read_json_by_line will be true.

Behavior patterns after this PR:

1、In scenarios such as S3 load, since read_json_by_line is not only
related to importing JSON formats but also serves as the switch for
streaming JSON file reading, it will be hardcoded to true (thus, JSON
formats requiring this parameter to be false are not supported in such
environments).
2、In scenarios such as Stream Load, users have absolute freedom to
specify any combination of parameter values (though typically we do not
expect users to actively set either to false).

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot requested a review from morrySnow as a code owner October 9, 2025 05:49
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Oct 9, 2025
@dataroaring dataroaring reopened this Oct 9, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

TPC-H: Total hot run time: 32904 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d011538af2d1e9b69182107a544720751edafa8c, data reload: false

------ Round 1 ----------------------------------
q1	17582	5596	5378	5378
q2	2014	424	321	321
q3	11642	1230	785	785
q4	10268	866	460	460
q5	8699	2443	2104	2104
q6	181	165	134	134
q7	896	770	633	633
q8	9339	1446	1224	1224
q9	5251	4950	4955	4950
q10	6785	2273	1811	1811
q11	475	292	263	263
q12	337	362	228	228
q13	17801	3635	3035	3035
q14	222	222	210	210
q15	533	463	464	463
q16	420	439	367	367
q17	598	862	357	357
q18	6961	6447	6394	6394
q19	1390	952	562	562
q20	330	356	217	217
q21	2767	2212	2045	2045
q22	1056	1034	963	963
Total cold run time: 105547 ms
Total hot run time: 32904 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5643	5551	5534	5534
q2	241	323	235	235
q3	2268	2699	2400	2400
q4	1401	1860	1378	1378
q5	4469	4999	5007	4999
q6	174	162	134	134
q7	2149	1998	1875	1875
q8	2722	2959	2779	2779
q9	7251	7183	7213	7183
q10	3093	3335	2783	2783
q11	585	522	511	511
q12	688	748	632	632
q13	3455	3834	3242	3242
q14	284	300	289	289
q15	512	473	467	467
q16	444	492	442	442
q17	1279	1745	1259	1259
q18	7609	7447	7292	7292
q19	816	1226	1116	1116
q20	2047	2057	1901	1901
q21	5550	5013	4746	4746
q22	1136	1053	1021	1021
Total cold run time: 53816 ms
Total hot run time: 52218 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

TPC-DS: Total hot run time: 194013 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d011538af2d1e9b69182107a544720751edafa8c, data reload: false

query1	948	407	449	407
query2	6225	1998	1913	1913
query3	8694	200	194	194
query4	33750	24177	23870	23870
query5	3722	608	473	473
query6	293	201	186	186
query7	4206	507	313	313
query8	297	244	242	242
query9	9472	2607	2631	2607
query10	472	319	263	263
query11	18006	15393	15276	15276
query12	158	107	111	107
query13	1568	555	440	440
query14	9216	7968	7261	7261
query15	253	192	190	190
query16	8141	672	516	516
query17	1608	807	614	614
query18	2174	427	331	331
query19	229	209	175	175
query20	131	123	124	123
query21	210	131	115	115
query22	4633	4646	4453	4453
query23	35191	34358	33920	33920
query24	7766	2715	2784	2715
query25	563	510	440	440
query26	1151	290	178	178
query27	2390	531	383	383
query28	5051	2306	2262	2262
query29	820	613	475	475
query30	244	202	166	166
query31	1034	942	856	856
query32	89	67	63	63
query33	530	414	330	330
query34	774	873	532	532
query35	798	838	741	741
query36	1006	1063	997	997
query37	113	101	74	74
query38	4049	4069	4062	4062
query39	1551	1477	1480	1477
query40	207	127	112	112
query41	50	50	46	46
query42	138	106	105	105
query43	524	538	491	491
query44	1414	868	861	861
query45	193	178	178	178
query46	899	1073	694	694
query47	1968	1984	1922	1922
query48	427	431	347	347
query49	769	511	417	417
query50	712	709	450	450
query51	7432	7344	7273	7273
query52	105	109	99	99
query53	254	267	204	204
query54	565	576	486	486
query55	81	79	80	79
query56	286	288	258	258
query57	1264	1289	1225	1225
query58	244	221	228	221
query59	3031	3172	3072	3072
query60	300	292	284	284
query61	117	114	117	114
query62	815	755	703	703
query63	242	199	200	199
query64	4576	1053	659	659
query65	3404	3318	3324	3318
query66	1018	407	314	314
query67	16257	15781	15523	15523
query68	8000	857	561	561
query69	514	306	281	281
query70	1193	1153	1137	1137
query71	387	299	271	271
query72	5663	3872	3891	3872
query73	654	761	357	357
query74	10469	9188	9174	9174
query75	3237	3139	2687	2687
query76	3356	1152	758	758
query77	564	402	291	291
query78	10418	10399	9549	9549
query79	3243	910	619	619
query80	653	535	439	439
query81	513	253	223	223
query82	571	124	87	87
query83	174	162	142	142
query84	289	99	79	79
query85	796	356	305	305
query86	383	317	309	309
query87	4302	4322	4263	4263
query88	5343	2553	2423	2423
query89	409	330	301	301
query90	1869	194	195	194
query91	147	140	110	110
query92	70	57	53	53
query93	2121	930	570	570
query94	667	430	323	323
query95	352	285	288	285
query96	501	613	280	280
query97	3159	3235	3189	3189
query98	231	208	194	194
query99	1531	1425	1293	1293
Total cold run time: 296597 ms
Total hot run time: 194013 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2025

ClickBench: Total hot run time: 29.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d011538af2d1e9b69182107a544720751edafa8c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.53	0.50	0.52
query6	1.14	0.73	0.73
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.49	0.51
query10	0.54	0.54	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.66	0.60	0.59
query14	0.81	0.79	0.79
query15	0.84	0.84	0.82
query16	0.40	0.38	0.41
query17	1.05	1.02	0.98
query18	0.23	0.22	0.23
query19	1.98	1.83	1.90
query20	0.02	0.01	0.01
query21	15.37	0.96	0.58
query22	0.74	0.73	0.72
query23	15.11	1.39	0.62
query24	3.23	1.13	1.79
query25	0.18	0.09	0.08
query26	0.27	0.14	0.13
query27	0.04	0.05	0.04
query28	13.82	0.95	0.45
query29	12.62	3.99	3.21
query30	0.26	0.10	0.08
query31	2.83	0.63	0.39
query32	3.22	0.54	0.47
query33	3.06	3.03	3.00
query34	16.49	5.18	4.53
query35	4.56	4.61	4.61
query36	0.66	0.48	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.02 s
Total hot run time: 29.18 s

@morrySnow morrySnow merged commit 659a502 into branch-3.1 Oct 11, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-55861-branch-3.1 branch October 11, 2025 03:29
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @0AyanamiRei