[fix](txn insert) sub txn map is not thread safe #37265

mymeiyi · 2024-07-04T03:25:02Z

Proposed changes

The sub txn map in DatabaseTransactionMgr is not thread safe

doris-robot · 2024-07-04T03:25:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

mymeiyi · 2024-07-04T03:30:02Z

run buildall

gavinchou · 2024-07-04T03:55:29Z

fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java

- subTxnIdToTxnId.put(subTransactionId, transactionId);
+ writeLock();
+ try {
+ subTxnIdToTxnId.put(subTransactionId, transactionId);


subTxnIdToTxnId is a concurrent map, is it necessary to lock it?

it's not concurrent map private final Map<Long, Long> subTxnIdToTxnId = Maps.newHashMap()

doris-robot · 2024-07-04T04:52:27Z

TPC-H: Total hot run time: 39953 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efbdae766d2d70f9e1934d038a7f3fe824253579, data reload: false

------ Round 1 ----------------------------------
q1	17628	4481	4308	4308
q2	2016	190	187	187
q3	10442	1135	1134	1134
q4	10194	814	791	791
q5	7513	2668	2627	2627
q6	225	139	136	136
q7	955	598	599	598
q8	9230	2080	2069	2069
q9	9077	6531	6471	6471
q10	8941	3720	3712	3712
q11	451	241	241	241
q12	461	238	229	229
q13	18251	2961	2943	2943
q14	269	225	218	218
q15	512	482	495	482
q16	516	379	381	379
q17	984	645	731	645
q18	8087	7614	7441	7441
q19	3775	1450	1523	1450
q20	686	314	316	314
q21	4957	3233	3299	3233
q22	396	347	345	345
Total cold run time: 115566 ms
Total hot run time: 39953 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4237	4236	4236
q2	372	267	256	256
q3	2952	2766	2866	2766
q4	1980	1712	1731	1712
q5	5651	5497	5492	5492
q6	223	130	131	130
q7	2183	1826	1835	1826
q8	3300	3413	3418	3413
q9	8654	8588	8716	8588
q10	4071	3914	3796	3796
q11	575	483	487	483
q12	773	642	664	642
q13	17135	3182	3152	3152
q14	295	280	276	276
q15	536	489	488	488
q16	481	426	417	417
q17	1826	1545	1501	1501
q18	8189	7907	7808	7808
q19	1800	1564	1651	1564
q20	2071	1926	1837	1837
q21	5907	4906	4893	4893
q22	619	538	543	538
Total cold run time: 73950 ms
Total hot run time: 55814 ms

doris-robot · 2024-07-04T05:03:23Z

TPC-DS: Total hot run time: 173159 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efbdae766d2d70f9e1934d038a7f3fe824253579, data reload: false

query1	925	388	372	372
query2	6457	2428	2406	2406
query3	6634	213	213	213
query4	19669	17608	17577	17577
query5	3667	483	474	474
query6	255	179	169	169
query7	4587	301	292	292
query8	316	303	293	293
query9	8723	2456	2438	2438
query10	578	313	293	293
query11	10568	10086	9971	9971
query12	119	91	81	81
query13	1636	368	372	368
query14	10194	7059	7634	7059
query15	231	186	190	186
query16	7644	305	306	305
query17	1797	539	518	518
query18	1870	268	266	266
query19	201	144	156	144
query20	89	85	83	83
query21	212	127	128	127
query22	4411	4154	4026	4026
query23	33806	33632	33623	33623
query24	10942	2829	2838	2829
query25	624	368	370	368
query26	1063	156	167	156
query27	2272	327	331	327
query28	6508	2180	2178	2178
query29	909	651	641	641
query30	249	154	173	154
query31	1002	776	765	765
query32	97	53	52	52
query33	736	295	287	287
query34	1067	469	482	469
query35	784	633	633	633
query36	1145	1011	958	958
query37	147	80	83	80
query38	2940	2808	2796	2796
query39	885	862	825	825
query40	203	132	127	127
query41	62	54	53	53
query42	123	106	102	102
query43	589	573	570	570
query44	1191	756	733	733
query45	196	168	168	168
query46	1069	726	719	719
query47	1843	1746	1743	1743
query48	374	316	297	297
query49	853	409	418	409
query50	760	385	389	385
query51	6813	6774	6703	6703
query52	105	89	93	89
query53	357	287	294	287
query54	958	453	455	453
query55	77	75	73	73
query56	285	261	265	261
query57	1131	1050	1072	1050
query58	257	239	250	239
query59	3506	3249	3171	3171
query60	297	275	269	269
query61	93	114	112	112
query62	608	490	448	448
query63	323	297	292	292
query64	8664	2253	1734	1734
query65	3214	3085	3126	3085
query66	751	327	349	327
query67	15517	14857	15136	14857
query68	4474	550	555	550
query69	454	321	315	315
query70	1193	1194	1165	1165
query71	405	278	275	275
query72	6927	5114	5189	5114
query73	746	323	320	320
query74	5863	5584	5506	5506
query75	3387	2669	2682	2669
query76	2603	971	924	924
query77	425	309	305	305
query78	9558	8840	8963	8840
query79	2890	513	523	513
query80	2006	469	471	469
query81	599	219	223	219
query82	963	114	103	103
query83	298	171	170	170
query84	262	90	87	87
query85	1418	283	275	275
query86	473	339	319	319
query87	3322	3121	3053	3053
query88	3737	2380	2364	2364
query89	480	386	385	385
query90	1758	189	184	184
query91	136	100	102	100
query92	57	49	47	47
query93	2105	510	517	510
query94	1048	208	213	208
query95	409	315	329	315
query96	586	270	270	270
query97	3176	3001	3050	3001
query98	227	200	196	196
query99	1129	830	842	830
Total cold run time: 269292 ms
Total hot run time: 173159 ms

doris-robot · 2024-07-04T05:08:55Z

ClickBench: Total hot run time: 30.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efbdae766d2d70f9e1934d038a7f3fe824253579, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.08	0.08
query5	0.49	0.48	0.49
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.49
query10	0.55	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.57
query14	0.79	0.80	0.77
query15	0.85	0.82	0.81
query16	0.36	0.35	0.36
query17	1.02	0.99	1.03
query18	0.21	0.26	0.23
query19	1.82	1.78	1.77
query20	0.01	0.01	0.01
query21	15.40	0.79	0.66
query22	4.20	7.94	1.50
query23	18.29	1.46	1.25
query24	2.14	0.24	0.24
query25	0.16	0.08	0.09
query26	0.26	0.17	0.17
query27	0.09	0.08	0.07
query28	13.19	1.03	0.99
query29	12.65	3.31	3.27
query30	0.27	0.08	0.06
query31	2.86	0.39	0.39
query32	3.26	0.47	0.46
query33	2.92	2.89	2.96
query34	17.22	4.36	4.48
query35	4.54	4.46	4.46
query36	0.64	0.46	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.15
query41	0.09	0.06	0.06
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.58 s
Total hot run time: 30.17 s

dataroaring

LGTM

github-actions · 2024-07-04T10:07:27Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-04T10:07:28Z

PR approved by anyone and no changes requested.

mymeiyi · 2024-07-06T00:18:11Z

run buildall

doris-robot · 2024-07-06T00:57:41Z

TPC-H: Total hot run time: 39596 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cae335647114bc96ad037db22d658fcdd4fd8ab3, data reload: false

------ Round 1 ----------------------------------
q1	17627	4329	4238	4238
q2	2019	202	190	190
q3	10446	1191	1133	1133
q4	10198	716	726	716
q5	7471	2651	2548	2548
q6	215	138	141	138
q7	953	598	607	598
q8	9229	2063	2069	2063
q9	8995	6467	6479	6467
q10	8887	3686	3708	3686
q11	436	236	240	236
q12	556	236	226	226
q13	17833	3003	2981	2981
q14	253	226	222	222
q15	542	471	472	471
q16	516	375	365	365
q17	963	705	721	705
q18	8097	7528	7413	7413
q19	2874	1473	1428	1428
q20	651	309	321	309
q21	5289	3123	3930	3123
q22	399	346	340	340
Total cold run time: 114449 ms
Total hot run time: 39596 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4264	4285	4264
q2	354	271	258	258
q3	3000	2771	2824	2771
q4	1944	1680	1734	1680
q5	5510	5687	5484	5484
q6	218	133	138	133
q7	2251	1827	1873	1827
q8	3305	3397	3399	3397
q9	8729	8679	8793	8679
q10	4082	3950	3883	3883
q11	588	471	498	471
q12	812	632	649	632
q13	16322	3171	3176	3171
q14	302	282	273	273
q15	543	498	493	493
q16	507	435	429	429
q17	1832	1535	1498	1498
q18	8016	7874	7794	7794
q19	1811	1489	1737	1489
q20	2074	1839	1908	1839
q21	4986	5077	5075	5075
q22	639	562	571	562
Total cold run time: 72238 ms
Total hot run time: 56102 ms

doris-robot · 2024-07-06T01:08:59Z

TPC-DS: Total hot run time: 173893 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cae335647114bc96ad037db22d658fcdd4fd8ab3, data reload: false

query1	915	364	362	362
query2	6460	2451	2345	2345
query3	6629	200	211	200
query4	28206	17501	17249	17249
query5	3633	495	488	488
query6	278	171	168	168
query7	4581	295	289	289
query8	314	287	304	287
query9	8580	2387	2349	2349
query10	593	315	286	286
query11	11630	9954	9950	9950
query12	117	88	87	87
query13	1650	389	382	382
query14	10284	7795	7358	7358
query15	258	188	185	185
query16	7431	318	318	318
query17	1800	585	546	546
query18	1852	292	288	288
query19	203	156	153	153
query20	93	84	86	84
query21	213	147	140	140
query22	4228	4037	4049	4037
query23	34118	33738	33780	33738
query24	11030	2809	2846	2809
query25	628	421	413	413
query26	754	167	167	167
query27	2240	322	336	322
query28	6580	2101	2107	2101
query29	917	764	631	631
query30	246	158	159	158
query31	984	750	756	750
query32	91	56	57	56
query33	759	297	308	297
query34	1005	492	502	492
query35	738	618	682	618
query36	1132	989	990	989
query37	147	93	92	92
query38	3024	2948	2891	2891
query39	927	863	855	855
query40	219	134	136	134
query41	53	52	51	51
query42	120	99	105	99
query43	601	560	559	559
query44	1185	743	715	715
query45	192	155	155	155
query46	1083	759	716	716
query47	1844	1759	1781	1759
query48	368	296	286	286
query49	835	419	419	419
query50	781	395	383	383
query51	6891	6759	6797	6759
query52	111	93	97	93
query53	359	303	292	292
query54	936	447	459	447
query55	77	69	73	69
query56	299	269	274	269
query57	1123	1066	1082	1066
query58	265	245	248	245
query59	3503	3199	3261	3199
query60	322	277	298	277
query61	97	95	92	92
query62	624	426	467	426
query63	326	291	292	291
query64	9528	2191	1610	1610
query65	3241	3100	3111	3100
query66	742	328	328	328
query67	15623	14847	14996	14847
query68	4611	531	524	524
query69	693	441	374	374
query70	1248	1076	1136	1076
query71	449	284	283	283
query72	8884	5512	5642	5512
query73	756	323	322	322
query74	5879	5573	5465	5465
query75	3943	2657	2671	2657
query76	3257	913	893	893
query77	705	313	308	308
query78	9500	9149	9307	9149
query79	4475	526	521	521
query80	2411	495	484	484
query81	602	221	223	221
query82	1073	110	115	110
query83	284	174	175	174
query84	280	88	88	88
query85	1349	328	314	314
query86	452	324	312	312
query87	3297	3170	3089	3089
query88	4701	2349	2385	2349
query89	508	410	388	388
query90	1929	189	195	189
query91	135	104	106	104
query92	62	50	52	50
query93	5468	511	504	504
query94	1251	207	209	207
query95	411	311	319	311
query96	616	269	263	263
query97	3184	3028	3004	3004
query98	218	200	205	200
query99	1144	853	832	832
Total cold run time: 290391 ms
Total hot run time: 173893 ms

doris-robot · 2024-07-06T01:14:32Z

ClickBench: Total hot run time: 30.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cae335647114bc96ad037db22d658fcdd4fd8ab3, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.06	0.04
query4	1.68	0.07	0.07
query5	0.49	0.49	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.58	0.59	0.58
query14	0.76	0.77	0.78
query15	0.82	0.82	0.82
query16	0.36	0.37	0.37
query17	1.02	1.03	1.06
query18	0.23	0.25	0.27
query19	1.84	1.76	1.83
query20	0.01	0.00	0.01
query21	15.39	0.73	0.64
query22	4.03	7.76	1.95
query23	18.34	1.41	1.26
query24	2.19	0.22	0.22
query25	0.15	0.09	0.08
query26	0.28	0.20	0.21
query27	0.44	0.23	0.24
query28	13.27	1.00	1.00
query29	12.63	3.26	3.28
query30	0.26	0.06	0.06
query31	2.87	0.40	0.38
query32	3.27	0.48	0.47
query33	2.90	2.89	2.91
query34	17.27	4.33	4.35
query35	4.47	4.39	4.40
query36	0.65	0.49	0.49
query37	0.18	0.15	0.14
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.16	0.11	0.12
query41	0.10	0.06	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.89 s
Total hot run time: 30.68 s

dataroaring

LGTM

github-actions · 2024-07-06T08:39:52Z

PR approved by at least one committer and no changes requested.

## Proposed changes The sub txn map in DatabaseTransactionMgr is not thread safe

gavinchou reviewed Jul 4, 2024

View reviewed changes

dataroaring previously approved these changes Jul 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024

github-actions bot added the reviewed label Jul 4, 2024

sub txn map is not thread safe

cae3356

mymeiyi dismissed dataroaring’s stale review via cae3356 July 6, 2024 00:17

mymeiyi force-pushed the fix-sub-txn branch from efbdae7 to cae3356 Compare July 6, 2024 00:17

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 6, 2024

dataroaring approved these changes Jul 6, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 6, 2024

Yukang-Lian approved these changes Jul 6, 2024

View reviewed changes

dataroaring merged commit 8e365e8 into apache:master Jul 7, 2024
27 of 30 checks passed

dataroaring pushed a commit that referenced this pull request Jul 17, 2024

[fix](txn insert) sub txn map is not thread safe (#37265)

85f55cc

## Proposed changes The sub txn map in DatabaseTransactionMgr is not thread safe

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](txn insert) sub txn map is not thread safe #37265

[fix](txn insert) sub txn map is not thread safe #37265

mymeiyi commented Jul 4, 2024

doris-robot commented Jul 4, 2024

mymeiyi commented Jul 4, 2024

gavinchou Jul 4, 2024

mymeiyi Jul 4, 2024

doris-robot commented Jul 4, 2024

doris-robot commented Jul 4, 2024

doris-robot commented Jul 4, 2024

dataroaring left a comment

github-actions bot commented Jul 4, 2024

github-actions bot commented Jul 4, 2024

mymeiyi commented Jul 6, 2024

doris-robot commented Jul 6, 2024

doris-robot commented Jul 6, 2024

doris-robot commented Jul 6, 2024

dataroaring left a comment

github-actions bot commented Jul 6, 2024

[fix](txn insert) sub txn map is not thread safe #37265

[fix](txn insert) sub txn map is not thread safe #37265

Conversation

mymeiyi commented Jul 4, 2024

Proposed changes

doris-robot commented Jul 4, 2024

mymeiyi commented Jul 4, 2024

gavinchou Jul 4, 2024

Choose a reason for hiding this comment

mymeiyi Jul 4, 2024

Choose a reason for hiding this comment

doris-robot commented Jul 4, 2024

doris-robot commented Jul 4, 2024

doris-robot commented Jul 4, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 4, 2024

github-actions bot commented Jul 4, 2024

mymeiyi commented Jul 6, 2024

doris-robot commented Jul 6, 2024

doris-robot commented Jul 6, 2024

doris-robot commented Jul 6, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 6, 2024