[branch-2.0] Picks "[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331" #39078

bobhan1 · 2024-08-08T03:39:55Z

picks #38331 and #39066

doris-robot · 2024-08-08T03:40:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-08T03:48:23Z

be/src/olap/partial_update_info.h

 // under the License.

 #pragma once
+<<<<<<< HEAD


warning: version control conflict marker in file [clang-diagnostic-error]

<<<<<<< HEAD ^

github-actions · 2024-08-08T03:48:23Z

be/src/olap/rowset/rowset_meta_manager.h

 #ifndef DORIS_BE_SRC_OLAP_ROWSET_ROWSET_META_MANAGER_H
 #define DORIS_BE_SRC_OLAP_ROWSET_ROWSET_META_MANAGER_H

+#include <gen_cpp/olap_file.pb.h>


warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]

#include <gen_cpp/olap_file.pb.h> ^

github-actions · 2024-08-08T03:48:23Z

be/src/olap/rowset_builder.cpp

+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset_builder.h"


warning: 'olap/rowset_builder.h' file not found [clang-diagnostic-error]

#include "olap/rowset_builder.h" ^

bobhan1 · 2024-08-08T04:14:18Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-08T04:17:02Z

be/src/olap/partial_update_info.cpp

+
+#include "olap/partial_update_info.h"
+
+#include <gen_cpp/olap_file.pb.h>


warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]

#include <gen_cpp/olap_file.pb.h> ^

github-actions · 2024-08-08T04:17:03Z

be/src/olap/partial_update_info.h

+              int64_t timestamp_ms, const std::string& timezone, int64_t cur_max_version = -1);
+    void to_pb(PartialUpdateInfoPB* partial_update_info) const;
+    void from_pb(PartialUpdateInfoPB* partial_update_info);
+    std::string summary() const;


warning: function 'summary' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change

std::string summary() const;

[[nodiscard]] std::string summary() const;

bobhan1 · 2024-08-08T05:14:28Z

run buildall

tmp tmp tmp tmp finish finish case tmp fix fix complie tmp tmp tmp tmp fix deserilize fix compile fix compile and pick 39066 fix compile

bobhan1 · 2024-08-08T07:42:23Z

run buildall

bobhan1 · 2024-08-08T08:59:31Z

run buildall

bobhan1 · 2024-08-08T09:33:14Z

run buildall

bobhan1 · 2024-08-08T10:51:20Z

run buildall

doris-robot · 2024-08-08T11:19:58Z

TPC-H: Total hot run time: 50382 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e490ad38125c85d7d67f45c1def817feb9d676a4, data reload: false

------ Round 1 ----------------------------------
q1	17823	4466	4407	4407
q2	2108	158	149	149
q3	10335	1899	1904	1899
q4	10108	1260	1350	1260
q5	8488	3931	3986	3931
q6	266	140	126	126
q7	2039	1627	1606	1606
q8	9273	2797	2736	2736
q9	11175	10420	10617	10420
q10	8677	3579	3540	3540
q11	414	248	259	248
q12	473	307	302	302
q13	18362	3990	4025	3990
q14	358	338	333	333
q15	504	452	457	452
q16	685	582	574	574
q17	1150	974	997	974
q18	7308	6816	6903	6816
q19	1861	1689	1683	1683
q20	532	325	310	310
q21	4495	4182	4197	4182
q22	545	454	444	444
Total cold run time: 116979 ms
Total hot run time: 50382 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4324	4330	4324
q2	321	220	221	220
q3	4153	4189	4174	4174
q4	2756	2770	2768	2768
q5	7218	7184	7134	7134
q6	241	122	121	121
q7	3254	2847	2835	2835
q8	4386	4499	4495	4495
q9	17534	17360	17261	17261
q10	4229	4251	4269	4251
q11	768	705	680	680
q12	1029	845	877	845
q13	7367	3729	3769	3729
q14	458	425	440	425
q15	493	465	453	453
q16	743	672	678	672
q17	3846	3844	3834	3834
q18	8922	8805	8819	8805
q19	1765	1730	1670	1670
q20	2385	2115	2152	2115
q21	8578	8548	8483	8483
q22	1097	964	965	964
Total cold run time: 85881 ms
Total hot run time: 80258 ms

doris-robot · 2024-08-08T11:32:36Z

TPC-DS: Total hot run time: 202875 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e490ad38125c85d7d67f45c1def817feb9d676a4, data reload: false

query1	936	388	415	388
query2	6543	2817	2633	2633
query3	6923	206	198	198
query4	19946	18007	17969	17969
query5	19735	6548	6485	6485
query6	291	216	228	216
query7	4147	308	314	308
query8	416	445	384	384
query9	3169	2738	2682	2682
query10	409	301	293	293
query11	11248	10587	10715	10587
query12	129	78	81	78
query13	5606	693	704	693
query14	17727	13440	13199	13199
query15	368	245	244	244
query16	6467	287	264	264
query17	1733	1456	879	879
query18	2321	413	411	411
query19	209	150	148	148
query20	81	77	80	77
query21	190	101	96	96
query22	5291	4876	5025	4876
query23	32781	31908	31908	31908
query24	6936	6530	6562	6530
query25	541	424	421	421
query26	527	161	161	161
query27	1904	300	293	293
query28	6238	2413	2389	2389
query29	2936	2616	2899	2616
query30	238	169	165	165
query31	928	768	744	744
query32	73	65	55	55
query33	411	249	254	249
query34	870	484	488	484
query35	1131	978	951	951
query36	1169	1182	1081	1081
query37	88	59	62	59
query38	3077	2918	2929	2918
query39	1362	1318	1316	1316
query40	208	94	88	88
query41	46	43	44	43
query42	92	93	86	86
query43	645	703	683	683
query44	1163	723	730	723
query45	249	236	230	230
query46	1212	966	996	966
query47	1866	1722	1930	1722
query48	1017	738	716	716
query49	623	361	362	361
query50	871	629	611	611
query51	4770	4660	4627	4627
query52	94	82	80	80
query53	449	325	329	325
query54	2683	2470	2489	2470
query55	96	85	83	83
query56	222	224	211	211
query57	1191	1149	1103	1103
query58	209	204	201	201
query59	4241	3736	3776	3736
query60	225	206	198	198
query61	96	94	94	94
query62	859	488	456	456
query63	490	342	339	339
query64	2334	1534	1485	1485
query65	3635	3535	3569	3535
query66	806	378	379	378
query67	17498	15273	15164	15164
query68	8737	668	646	646
query69	572	342	334	334
query70	1542	1498	1369	1369
query71	408	291	314	291
query72	6694	3441	3516	3441
query73	744	321	322	321
query74	6356	5793	5882	5793
query75	5308	3729	3592	3592
query76	5306	1137	1197	1137
query77	898	256	253	253
query78	12758	11799	17930	11799
query79	6630	659	639	639
query80	904	407	411	407
query81	492	242	234	234
query82	269	102	99	99
query83	188	141	135	135
query84	255	70	70	70
query85	910	330	338	330
query86	325	300	311	300
query87	3219	3003	2997	2997
query88	3915	2330	2313	2313
query89	346	306	314	306
query90	1973	192	214	192
query91	177	148	139	139
query92	57	53	52	52
query93	1026	571	526	526
query94	697	205	207	205
query95	1102	1047	1056	1047
query96	630	329	327	327
query97	6501	6437	6526	6437
query98	176	172	178	172
query99	3033	847	810	810
Total cold run time: 305133 ms
Total hot run time: 202875 ms

doris-robot · 2024-08-08T11:38:03Z

ClickBench: Total hot run time: 30.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e490ad38125c85d7d67f45c1def817feb9d676a4, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.77	0.08	0.08
query5	0.54	0.53	0.51
query6	1.24	0.60	0.60
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.54	0.51	0.48
query10	0.54	0.54	0.53
query11	0.12	0.08	0.08
query12	0.12	0.10	0.09
query13	0.62	0.62	0.61
query14	0.79	0.78	0.77
query15	0.78	0.76	0.76
query16	0.36	0.37	0.37
query17	1.02	1.03	1.02
query18	0.22	0.25	0.26
query19	1.94	1.86	1.86
query20	0.01	0.01	0.01
query21	15.47	0.60	0.56
query22	2.07	2.36	1.99
query23	17.24	0.93	0.96
query24	5.49	0.88	1.01
query25	0.37	0.07	0.08
query26	0.66	0.16	0.15
query27	0.04	0.04	0.03
query28	7.71	0.75	0.69
query29	12.62	2.42	2.14
query30	0.59	0.56	0.51
query31	2.83	0.37	0.38
query32	3.40	0.50	0.49
query33	3.06	3.11	3.06
query34	15.27	4.83	4.79
query35	4.91	4.86	4.86
query36	1.05	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.16 s
Total hot run time: 30.93 s

doris-robot · 2024-08-08T11:41:19Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e490ad38125c85d7d67f45c1def817feb9d676a4 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

doris-robot · 2024-08-08T11:50:38Z

TeamCity be ut coverage result:
Function Coverage: 37.72% (8103/21482)
Line Coverage: 29.36% (66401/226179)
Region Coverage: 28.85% (34247/118687)
Branch Coverage: 24.74% (17603/71152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e490ad38125c85d7d67f45c1def817feb9d676a4_e490ad38125c85d7d67f45c1def817feb9d676a4/report/index.html

… in RocksDB in case of BE restart after a partial update has commited apache#38331" (apache#39078) picks apache#38331 and apache#39066

github-actions bot added the kind/test label Aug 8, 2024

bobhan1 mentioned this pull request Aug 8, 2024

[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331

Merged

github-actions bot reviewed Aug 8, 2024

View reviewed changes

bobhan1 force-pushed the branch-2.0-pick-38331 branch from f97963f to df9b0b3 Compare August 8, 2024 04:13

github-actions bot reviewed Aug 8, 2024

View reviewed changes

bobhan1 force-pushed the branch-2.0-pick-38331 branch 2 times, most recently from b365273 to 48f911e Compare August 8, 2024 05:12

bobhan1 added 2 commits August 8, 2024 15:39

update

40fcbc4

tmp tmp tmp tmp finish finish case tmp fix fix complie tmp tmp tmp tmp fix deserilize fix compile fix compile and pick 39066 fix compile

tmp

23a30f3

bobhan1 force-pushed the branch-2.0-pick-38331 branch from 48f911e to 23a30f3 Compare August 8, 2024 07:40

bobhan1 force-pushed the branch-2.0-pick-38331 branch from f93c84f to edaea0b Compare August 8, 2024 09:32

fix compile

e490ad3

bobhan1 force-pushed the branch-2.0-pick-38331 branch from edaea0b to e490ad3 Compare August 8, 2024 10:47

dataroaring merged commit bf11d48 into apache:branch-2.0 Aug 9, 2024


		#include "olap/partial_update_info.h"

		#include <gen_cpp/olap_file.pb.h>

	std::string summary() const;
	[[nodiscard]] std::string summary() const;

[branch-2.0] Picks "[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331" #39078

[branch-2.0] Picks "[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331" #39078

Uh oh!

Conversation

bobhan1 commented Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

bobhan1 commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bobhan1 commented Aug 8, 2024 •

edited

Loading