Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](ES Catalog)Support control scroll level by config #37180 #37271

Merged
merged 1 commit into from
Jul 15, 2024
Merged

[feature](ES Catalog)Support control scroll level by config #37180 #37271

merged 1 commit into from
Jul 15, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Jul 4, 2024

backport #37180

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Jul 4, 2024
@qidaye
Copy link
Contributor Author

qidaye commented Jul 4, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49597 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a878f6017dc9050a985facb00d81d93407ee5260, data reload: false

------ Round 1 ----------------------------------
q1	17781	4350	4302	4302
q2	2102	155	152	152
q3	10520	1929	1938	1929
q4	10391	1253	1306	1253
q5	8468	3934	3859	3859
q6	254	125	126	125
q7	2049	1609	1605	1605
q8	9495	2746	2701	2701
q9	13832	10254	10171	10171
q10	8639	3531	3542	3531
q11	401	242	249	242
q12	476	305	313	305
q13	18371	3945	4003	3945
q14	366	322	331	322
q15	497	464	466	464
q16	683	566	574	566
q17	1118	948	899	899
q18	7258	6892	6921	6892
q19	1809	1688	1551	1551
q20	571	306	293	293
q21	4440	4077	4058	4058
q22	542	452	432	432
Total cold run time: 120063 ms
Total hot run time: 49597 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4305	4271	4271
q2	319	226	244	226
q3	4121	4081	4122	4081
q4	2730	2731	2720	2720
q5	7172	7072	7087	7072
q6	240	119	117	117
q7	3222	2821	2844	2821
q8	4341	4441	4473	4441
q9	16959	16780	16765	16765
q10	4256	4268	4290	4268
q11	746	676	657	657
q12	1041	862	870	862
q13	7380	3702	3724	3702
q14	452	428	420	420
q15	500	454	450	450
q16	726	682	664	664
q17	3806	3762	3827	3762
q18	8738	8665	8798	8665
q19	1750	1652	1629	1629
q20	2359	2132	2115	2115
q21	8505	8255	8326	8255
q22	1063	981	952	952
Total cold run time: 84727 ms
Total hot run time: 78915 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.90% (8116/21417)
Line Coverage: 29.56% (66496/224930)
Region Coverage: 29.03% (34275/118055)
Branch Coverage: 24.90% (17604/70690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a878f6017dc9050a985facb00d81d93407ee5260_a878f6017dc9050a985facb00d81d93407ee5260/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202590 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a878f6017dc9050a985facb00d81d93407ee5260, data reload: false

query1	926	420	377	377
query2	6554	2836	2646	2646
query3	6927	202	200	200
query4	21426	17981	17974	17974
query5	19746	6425	6483	6425
query6	288	217	225	217
query7	4162	290	305	290
query8	439	405	459	405
query9	3081	2652	2582	2582
query10	400	287	300	287
query11	11374	10834	10731	10731
query12	125	83	72	72
query13	5588	669	684	669
query14	17837	13547	13557	13547
query15	357	237	251	237
query16	6457	281	262	262
query17	1701	1455	875	875
query18	2321	408	409	408
query19	205	150	153	150
query20	77	80	82	80
query21	189	92	93	92
query22	5152	5058	4851	4851
query23	32295	31846	31680	31680
query24	6932	6424	6453	6424
query25	534	416	441	416
query26	632	161	157	157
query27	2040	297	289	289
query28	6168	2331	2284	2284
query29	2904	2723	2771	2723
query30	242	165	168	165
query31	930	725	732	725
query32	72	59	60	59
query33	408	253	255	253
query34	835	475	464	464
query35	1114	920	888	888
query36	1260	1111	1168	1111
query37	93	62	60	60
query38	3058	2903	2913	2903
query39	1358	1333	1314	1314
query40	208	96	94	94
query41	46	44	44	44
query42	83	91	84	84
query43	765	677	682	677
query44	1102	706	716	706
query45	243	236	232	232
query46	1205	952	985	952
query47	1899	1642	1694	1642
query48	1014	694	712	694
query49	614	361	387	361
query50	861	625	575	575
query51	4689	4631	4693	4631
query52	91	77	76	76
query53	446	323	318	318
query54	2634	2401	2437	2401
query55	94	78	84	78
query56	234	226	210	210
query57	1203	1122	1130	1122
query58	229	208	191	191
query59	4422	3927	3845	3845
query60	215	196	201	196
query61	101	95	111	95
query62	918	499	426	426
query63	479	342	336	336
query64	2490	1467	1437	1437
query65	3601	3551	3516	3516
query66	816	377	377	377
query67	16061	15348	15755	15348
query68	8733	649	651	649
query69	553	342	352	342
query70	1571	1539	1308	1308
query71	413	304	319	304
query72	6559	3466	3458	3458
query73	741	318	328	318
query74	6291	5924	5818	5818
query75	5184	3658	3719	3658
query76	5173	1142	1206	1142
query77	871	251	263	251
query78	12633	11846	11851	11846
query79	8953	639	649	639
query80	1285	394	406	394
query81	495	237	232	232
query82	1458	98	95	95
query83	180	131	132	131
query84	252	70	70	70
query85	909	317	313	313
query86	341	288	312	288
query87	3280	2980	3010	2980
query88	4944	2316	2283	2283
query89	362	311	284	284
query90	1917	212	216	212
query91	176	143	138	138
query92	58	51	56	51
query93	4581	583	563	563
query94	748	217	214	214
query95	1124	1069	1043	1043
query96	631	324	328	324
query97	6504	6280	6315	6280
query98	181	187	176	176
query99	2960	907	830	830
Total cold run time: 313031 ms
Total hot run time: 202590 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a878f6017dc9050a985facb00d81d93407ee5260, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.81	0.07	0.06
query5	0.53	0.53	0.52
query6	1.23	0.66	0.63
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.54	0.49	0.48
query10	0.54	0.53	0.54
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.62	0.61	0.61
query14	0.78	0.77	0.79
query15	0.78	0.76	0.76
query16	0.35	0.36	0.35
query17	0.97	0.96	1.01
query18	0.19	0.28	0.23
query19	1.91	1.83	1.82
query20	0.02	0.01	0.01
query21	15.45	0.56	0.56
query22	2.11	1.84	1.86
query23	17.08	1.09	1.09
query24	3.85	4.72	2.19
query25	0.40	0.11	0.05
query26	0.89	0.18	0.14
query27	0.05	0.05	0.04
query28	4.46	0.74	0.72
query29	12.60	2.28	2.22
query30	0.58	0.52	0.53
query31	2.80	0.39	0.38
query32	3.38	0.50	0.50
query33	3.09	3.06	3.10
query34	15.28	4.82	4.77
query35	4.84	4.83	4.83
query36	1.06	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.01	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 99.18 s
Total hot run time: 32.18 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a878f6017dc9050a985facb00d81d93407ee5260 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 4, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50178 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7be5dddc8358107c9caef6f3e6ed67fe77f7d14d, data reload: false

------ Round 1 ----------------------------------
q1	17532	4348	4330	4330
q2	2077	160	147	147
q3	10259	1898	1989	1898
q4	10093	1275	1348	1275
q5	8428	3901	3931	3901
q6	237	126	125	125
q7	2074	1593	1626	1593
q8	9310	2761	2735	2735
q9	10740	10961	10680	10680
q10	8681	3517	3566	3517
q11	428	251	249	249
q12	466	310	307	307
q13	18382	3946	4055	3946
q14	345	328	329	328
q15	509	459	465	459
q16	673	563	560	560
q17	1152	954	964	954
q18	7246	6763	6853	6763
q19	1799	1690	1578	1578
q20	527	318	307	307
q21	4492	4087	4111	4087
q22	548	447	439	439
Total cold run time: 115998 ms
Total hot run time: 50178 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4335	4278	4271	4271
q2	324	220	224	220
q3	4198	4159	4135	4135
q4	2761	2758	2760	2758
q5	7168	7109	7160	7109
q6	235	119	119	119
q7	3235	2855	2897	2855
q8	4389	4538	4526	4526
q9	16943	16656	16748	16656
q10	4232	4244	4313	4244
q11	763	672	707	672
q12	1037	851	867	851
q13	4583	3785	3775	3775
q14	465	441	428	428
q15	513	452	449	449
q16	730	681	674	674
q17	3846	3811	3837	3811
q18	8849	8803	8704	8704
q19	1757	1757	1707	1707
q20	2406	2112	2106	2106
q21	8625	8439	8602	8439
q22	1080	1005	976	976
Total cold run time: 82474 ms
Total hot run time: 79485 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.89% (8114/21417)
Line Coverage: 29.55% (66479/224963)
Region Coverage: 29.02% (34261/118074)
Branch Coverage: 24.88% (17595/70708)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7be5dddc8358107c9caef6f3e6ed67fe77f7d14d_7be5dddc8358107c9caef6f3e6ed67fe77f7d14d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 205555 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7be5dddc8358107c9caef6f3e6ed67fe77f7d14d, data reload: false

query1	964	424	380	380
query2	6524	2758	2895	2758
query3	6925	206	204	204
query4	20196	18096	18060	18060
query5	19731	6581	6581	6581
query6	297	224	230	224
query7	4183	297	308	297
query8	413	483	431	431
query9	3128	2685	2634	2634
query10	419	292	295	292
query11	11530	11039	10990	10990
query12	127	81	68	68
query13	5601	684	692	684
query14	17943	13130	13191	13130
query15	373	246	246	246
query16	6470	285	258	258
query17	1745	1463	885	885
query18	2289	419	414	414
query19	210	148	150	148
query20	83	78	78	78
query21	187	90	90	90
query22	5347	5157	5124	5124
query23	32495	31851	31950	31851
query24	6917	6564	6480	6480
query25	528	424	422	422
query26	527	164	163	163
query27	1901	301	298	298
query28	6199	2392	2334	2334
query29	2874	2770	2762	2762
query30	243	168	168	168
query31	922	740	781	740
query32	69	56	60	56
query33	391	261	260	260
query34	849	487	472	472
query35	1107	922	915	915
query36	1368	1183	1252	1183
query37	89	60	59	59
query38	3068	2915	2900	2900
query39	1386	1354	1315	1315
query40	211	93	93	93
query41	46	43	44	43
query42	80	83	85	83
query43	775	726	666	666
query44	1126	709	709	709
query45	248	236	239	236
query46	1222	949	978	949
query47	1845	1776	1728	1728
query48	1015	745	713	713
query49	612	372	372	372
query50	869	602	577	577
query51	4828	4717	4655	4655
query52	81	81	80	80
query53	456	338	318	318
query54	2661	2430	2476	2430
query55	105	80	89	80
query56	227	231	203	203
query57	1178	1106	1078	1078
query58	216	227	195	195
query59	4201	4354	3814	3814
query60	207	192	216	192
query61	95	103	91	91
query62	832	541	495	495
query63	497	357	351	351
query64	2569	1542	1328	1328
query65	3601	3537	3538	3537
query66	812	378	375	375
query67	17264	15482	15063	15063
query68	10256	663	647	647
query69	567	358	353	353
query70	1848	1386	1460	1386
query71	409	315	310	310
query72	6509	3469	3501	3469
query73	723	322	317	317
query74	6444	5901	5893	5893
query75	5294	3713	3709	3709
query76	6544	1138	1220	1138
query77	1138	263	265	263
query78	12917	13427	13603	13427
query79	8265	646	665	646
query80	723	387	390	387
query81	493	237	237	237
query82	248	99	97	97
query83	174	136	136	136
query84	266	71	69	69
query85	842	321	337	321
query86	330	292	303	292
query87	3249	3055	3034	3034
query88	4759	2279	2286	2279
query89	377	292	292	292
query90	2647	207	212	207
query91	175	139	138	138
query92	58	52	51	51
query93	997	529	562	529
query94	1230	207	213	207
query95	1104	1067	1050	1050
query96	643	321	328	321
query97	6495	6405	6503	6405
query98	194	169	173	169
query99	2954	968	1001	968
Total cold run time: 312369 ms
Total hot run time: 205555 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7be5dddc8358107c9caef6f3e6ed67fe77f7d14d, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.03	0.03
query3	0.25	0.05	0.04
query4	1.78	0.06	0.07
query5	0.55	0.52	0.52
query6	1.23	0.61	0.62
query7	0.01	0.00	0.00
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.52
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.76	0.80	0.77
query15	0.78	0.76	0.77
query16	0.39	0.37	0.39
query17	1.05	1.00	1.02
query18	0.24	0.27	0.22
query19	1.87	1.84	1.82
query20	0.01	0.01	0.01
query21	15.49	0.54	0.54
query22	2.01	3.38	1.89
query23	17.10	1.11	0.89
query24	5.94	0.89	1.81
query25	0.33	0.10	0.04
query26	0.71	0.15	0.15
query27	0.06	0.03	0.04
query28	6.89	0.70	0.70
query29	12.71	2.42	2.19
query30	0.58	0.50	0.50
query31	2.81	0.38	0.37
query32	3.37	0.50	0.49
query33	3.08	3.07	3.05
query34	15.25	4.79	4.80
query35	4.81	4.86	4.82
query36	1.04	1.00	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.03	0.01	0.02
query40	0.15	0.14	0.15
query41	0.06	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.51 s
Total hot run time: 30.63 s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run external

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7be5dddc8358107c9caef6f3e6ed67fe77f7d14d with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run external

5 similar comments
@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run external

@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run external

@qidaye
Copy link
Contributor Author

qidaye commented Jul 8, 2024

run external

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run external

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run external

@morningman morningman merged commit 4acc692 into apache:branch-2.0 Jul 15, 2024
22 of 24 checks passed
@qidaye qidaye deleted the pick_scroll_config_2.0 branch July 15, 2024 08:42
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye
[feature](ES Catalog)Support control scroll level by config apache#37180
fcaf20c 
 (apache#37271)

backport apache#37180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qidaye @doris-robot @morningman