Skip to content

branch-3.1: [fix](csv reader) fix data loss when concurrency read using multi char line delimiter #53374 #53424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 17, 2025
Merged

branch-3.1: [fix](csv reader) fix data loss when concurrency read using multi char line delimiter #53374 #53424

morrySnow merged 1 commit into from
Jul 17, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 17, 2025

Cherry-picked from #53374

@sollhui
[fix](csv reader) fix data loss when concurrency read using multi cha…
5f2fd17 
…r line delimiter (#53374)

Multiple concurrent split file locations will be determined in plan
phase, if the split point happens to be in the middle of the multi char
line delimiter:

- The previous concurrent will read the complete row1 and read a little
more to read the line delimiter.
- The latter concurrency will start reading from half of the multi char
line delimiter, and row2 is the first line of this concurrency, but the
first line in the middle range is always discarded, so row2 will be
lost.
@github-actions github-actions bot requested a review from morrySnow as a code owner July 17, 2025 03:43
@Thearas
Copy link
Contributor

Thearas commented Jul 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 17, 2025
@Thearas
Copy link
Contributor

Thearas commented Jul 17, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 17, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.24% (12540/27718)
Line Coverage 36.10% (111386/308587)
Region Coverage 35.17% (57608/163813)
Branch Coverage 32.35% (31314/96806)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 17, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.09% (20836/27385)
Line Coverage 69.45% (214134/308349)
Region Coverage 67.46% (128084/189855)
Branch Coverage 61.07% (66685/109192)

@doris-robot
Copy link

doris-robot commented Jul 17, 2025

TPC-H: Total hot run time: 39921 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f2fd176901f1c9a76a9841c5d55096f60b31e2a, data reload: false

------ Round 1 ----------------------------------
q1	17637	6781	6640	6640
q2	2060	198	174	174
q3	10504	1155	1176	1155
q4	10225	794	764	764
q5	7757	3128	2875	2875
q6	224	138	141	138
q7	1008	622	626	622
q8	9371	2008	2038	2008
q9	6625	6393	6464	6393
q10	7016	2288	2312	2288
q11	452	264	263	263
q12	398	215	217	215
q13	17787	3035	2974	2974
q14	233	207	207	207
q15	511	471	472	471
q16	459	382	375	375
q17	979	575	564	564
q18	7331	6823	6592	6592
q19	1308	1031	1022	1022
q20	496	210	206	206
q21	3938	3126	3001	3001
q22	1094	974	982	974
Total cold run time: 107413 ms
Total hot run time: 39921 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6677	6637	6596	6596
q2	327	241	235	235
q3	2942	3007	2968	2968
q4	2058	1827	1748	1748
q5	5750	5691	5754	5691
q6	211	133	128	128
q7	2188	1808	1804	1804
q8	3364	3544	3514	3514
q9	8954	8873	8844	8844
q10	3582	3541	3542	3541
q11	590	512	501	501
q12	797	608	625	608
q13	6264	3210	3107	3107
q14	293	274	257	257
q15	513	473	466	466
q16	484	445	433	433
q17	1852	1622	1606	1606
q18	8172	7677	7716	7677
q19	1733	1463	1608	1463
q20	2141	1897	1889	1889
q21	5207	5036	4888	4888
q22	1117	992	978	978
Total cold run time: 65216 ms
Total hot run time: 58942 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2025

TPC-DS: Total hot run time: 191334 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f2fd176901f1c9a76a9841c5d55096f60b31e2a, data reload: false

query1	968	367	361	361
query2	6545	1910	1852	1852
query3	6701	218	237	218
query4	33734	23497	23592	23497
query5	4310	461	481	461
query6	272	173	181	173
query7	4624	313	334	313
query8	283	229	227	227
query9	9607	2611	2608	2608
query10	507	268	256	256
query11	18205	15274	15481	15274
query12	153	102	101	101
query13	1642	408	403	403
query14	9731	7190	7285	7190
query15	246	171	184	171
query16	8092	460	486	460
query17	1633	601	566	566
query18	2129	319	319	319
query19	248	169	164	164
query20	119	111	119	111
query21	211	105	105	105
query22	4512	4388	4130	4130
query23	34214	33556	33901	33556
query24	11763	2947	2891	2891
query25	711	409	424	409
query26	1751	177	170	170
query27	2903	350	334	334
query28	7782	2130	2118	2118
query29	1034	466	458	458
query30	327	165	158	158
query31	1007	803	843	803
query32	103	60	60	60
query33	795	318	317	317
query34	911	510	529	510
query35	869	709	701	701
query36	1090	986	939	939
query37	147	79	70	70
query38	3962	3829	3856	3829
query39	1487	1462	1428	1428
query40	293	104	103	103
query41	56	54	52	52
query42	118	105	105	105
query43	541	504	485	485
query44	1325	811	819	811
query45	187	174	173	173
query46	1139	724	746	724
query47	1927	1844	1821	1821
query48	444	353	366	353
query49	1282	412	417	412
query50	832	424	432	424
query51	7254	7254	7105	7105
query52	106	98	92	92
query53	266	191	189	189
query54	1251	493	487	487
query55	85	81	80	80
query56	293	272	271	271
query57	1302	1168	1195	1168
query58	251	230	229	229
query59	3045	2831	2833	2831
query60	300	281	280	280
query61	135	171	132	132
query62	872	674	699	674
query63	231	204	200	200
query64	5398	650	633	633
query65	3290	3228	3196	3196
query66	1438	321	309	309
query67	15980	15494	15596	15494
query68	4650	583	592	583
query69	442	289	266	266
query70	1201	1114	1094	1094
query71	336	261	267	261
query72	6351	4148	4054	4054
query73	760	346	366	346
query74	10645	9114	9109	9109
query75	3378	2676	2653	2653
query76	3066	1173	1088	1088
query77	397	288	281	281
query78	10446	9682	9577	9577
query79	2395	608	628	608
query80	1143	433	431	431
query81	539	223	219	219
query82	662	89	91	89
query83	217	147	151	147
query84	243	83	81	81
query85	1690	328	306	306
query86	476	305	312	305
query87	4404	4236	4266	4236
query88	4380	2419	2406	2406
query89	417	305	306	305
query90	2001	190	190	190
query91	139	111	110	110
query92	64	53	52	52
query93	1493	560	570	560
query94	878	288	307	288
query95	364	259	260	259
query96	617	288	294	288
query97	3334	3154	3168	3154
query98	224	206	202	202
query99	1509	1317	1242	1242
Total cold run time: 303830 ms
Total hot run time: 191334 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2025

ClickBench: Total hot run time: 30.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f2fd176901f1c9a76a9841c5d55096f60b31e2a, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.53	0.51	0.51
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.04	0.05	0.03
query9	0.56	0.49	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.10
query12	0.14	0.11	0.10
query13	0.61	0.61	0.60
query14	0.78	0.77	0.83
query15	0.85	0.86	0.83
query16	0.37	0.38	0.39
query17	1.05	1.01	1.04
query18	0.24	0.22	0.22
query19	1.95	1.87	1.84
query20	0.01	0.01	0.01
query21	15.40	0.60	0.58
query22	2.34	2.23	2.21
query23	17.01	0.96	0.86
query24	3.10	1.14	1.51
query25	0.12	0.23	0.14
query26	0.47	0.14	0.13
query27	0.04	0.04	0.04
query28	9.79	0.48	0.46
query29	12.58	3.18	3.17
query30	0.25	0.06	0.06
query31	2.85	0.40	0.40
query32	3.22	0.46	0.46
query33	3.03	2.98	2.99
query34	16.91	4.48	4.54
query35	4.49	4.50	4.54
query36	0.67	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.64 s
Total hot run time: 30.78 s

@morrySnow morrySnow merged commit eca9af8 into branch-3.1 Jul 17, 2025
20 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-53374-branch-3.1 branch July 17, 2025 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @hello-stephen @doris-robot @morrySnow @dataroaring @sollhui