Skip to content

[Enhancement] (nereids) implement alterRoutineLoadCommand in nereids #49960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

yx-keith
Copy link
Contributor

What problem does this PR solve?

Issue Number: #42795

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yx-keith
Copy link
Contributor Author

yx-keith commented Jun 9, 2025

run buildall

@yx-keith
Copy link
Contributor Author

yx-keith commented Jun 9, 2025

run buildall

@yx-keith
Copy link
Contributor Author

yx-keith commented Jun 9, 2025

run buildall

@yx-keith
Copy link
Contributor Author

yx-keith commented Jun 9, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33623 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 429ab846ca49985c7f4121a0abc7402d021f8a6f, data reload: false

------ Round 1 ----------------------------------
q1	25726	5005	4924	4924
q2	1940	271	172	172
q3	10330	1208	680	680
q4	10218	981	524	524
q5	7505	2344	2292	2292
q6	174	159	134	134
q7	902	719	606	606
q8	9291	1206	1041	1041
q9	6779	5069	5020	5020
q10	6849	2298	1871	1871
q11	503	294	271	271
q12	341	344	209	209
q13	17797	3696	3103	3103
q14	222	227	217	217
q15	578	499	504	499
q16	445	465	395	395
q17	619	867	372	372
q18	7934	7355	7253	7253
q19	1489	933	523	523
q20	323	333	225	225
q21	3684	3165	2335	2335
q22	1061	994	957	957
Total cold run time: 114710 ms
Total hot run time: 33623 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5058	5033	5019	5019
q2	232	315	216	216
q3	2156	2632	2261	2261
q4	1347	1750	1375	1375
q5	4439	4331	4423	4331
q6	216	171	129	129
q7	1965	1874	1739	1739
q8	2587	2550	2493	2493
q9	7188	7239	7014	7014
q10	3032	3198	2723	2723
q11	575	520	488	488
q12	682	758	603	603
q13	3488	3910	3316	3316
q14	284	300	282	282
q15	546	508	497	497
q16	456	492	439	439
q17	1124	1483	1399	1399
q18	7786	7507	7341	7341
q19	797	804	775	775
q20	2005	2067	1892	1892
q21	4726	4445	4365	4365
q22	1082	1047	993	993
Total cold run time: 51771 ms
Total hot run time: 49690 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191931 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 429ab846ca49985c7f4121a0abc7402d021f8a6f, data reload: false

query1	1446	1096	1044	1044
query2	6447	1847	1820	1820
query3	11004	4622	4333	4333
query4	56487	25469	23337	23337
query5	4903	472	459	459
query6	319	193	186	186
query7	4875	494	287	287
query8	317	218	204	204
query9	5768	2612	2621	2612
query10	434	340	265	265
query11	15064	14943	14780	14780
query12	154	106	103	103
query13	1038	519	438	438
query14	10179	6277	6193	6193
query15	206	215	172	172
query16	7190	650	481	481
query17	1090	764	598	598
query18	1731	431	323	323
query19	201	200	185	185
query20	149	122	120	120
query21	212	127	110	110
query22	4447	4467	4202	4202
query23	34208	33422	33690	33422
query24	6683	2389	2424	2389
query25	480	476	421	421
query26	686	271	150	150
query27	2246	501	342	342
query28	3042	2139	2150	2139
query29	584	563	450	450
query30	269	225	193	193
query31	852	864	756	756
query32	68	62	62	62
query33	439	365	314	314
query34	764	878	510	510
query35	804	812	761	761
query36	940	982	896	896
query37	134	99	75	75
query38	4268	4379	4193	4193
query39	1485	1472	1462	1462
query40	213	122	106	106
query41	60	61	55	55
query42	127	107	110	107
query43	503	520	481	481
query44	1337	856	879	856
query45	186	183	167	167
query46	866	1022	645	645
query47	1869	1943	1814	1814
query48	385	436	324	324
query49	640	493	393	393
query50	653	689	406	406
query51	4228	4234	4267	4234
query52	113	109	97	97
query53	235	258	187	187
query54	571	569	522	522
query55	83	81	83	81
query56	292	311	305	305
query57	1210	1234	1184	1184
query58	263	257	255	255
query59	2662	2729	2631	2631
query60	343	332	324	324
query61	129	121	140	121
query62	744	754	678	678
query63	234	194	193	193
query64	1762	1040	761	761
query65	4239	4125	4142	4125
query66	685	385	298	298
query67	15966	15673	15565	15565
query68	7083	868	534	534
query69	526	313	321	313
query70	1181	1028	1118	1028
query71	461	320	289	289
query72	5781	4809	4965	4809
query73	1361	679	357	357
query74	9267	9015	8660	8660
query75	3521	3210	2707	2707
query76	3743	1178	777	777
query77	559	393	285	285
query78	10142	10347	9348	9348
query79	1598	781	578	578
query80	648	509	510	509
query81	497	250	225	225
query82	220	122	98	98
query83	251	245	233	233
query84	303	105	93	93
query85	738	351	311	311
query86	349	305	274	274
query87	4363	4510	4283	4283
query88	2830	2300	2264	2264
query89	395	308	279	279
query90	1873	207	209	207
query91	154	134	112	112
query92	76	61	55	55
query93	1489	919	591	591
query94	652	405	312	312
query95	368	294	286	286
query96	490	569	284	284
query97	2689	2823	2666	2666
query98	222	205	205	205
query99	1313	1399	1257	1257
Total cold run time: 297354 ms
Total hot run time: 191931 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 429ab846ca49985c7f4121a0abc7402d021f8a6f, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.20	0.19
query4	1.60	0.20	0.20
query5	0.46	0.46	0.45
query6	1.16	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.57	0.57	0.57
query11	0.16	0.12	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.60
query14	0.81	0.80	0.79
query15	0.88	0.85	0.88
query16	0.39	0.39	0.37
query17	1.03	1.07	1.06
query18	0.24	0.21	0.21
query19	1.91	1.87	1.82
query20	0.01	0.01	0.01
query21	15.42	0.89	0.54
query22	0.76	1.37	0.71
query23	14.75	1.40	0.65
query24	6.80	1.48	0.29
query25	0.28	0.10	0.10
query26	0.62	0.17	0.15
query27	0.06	0.04	0.04
query28	9.61	0.94	0.43
query29	12.57	4.11	3.48
query30	0.26	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.53	0.48
query33	3.04	3.12	3.11
query34	15.88	5.15	4.45
query35	4.52	4.50	4.45
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 102.81 s
Total hot run time: 28.8 s

@yx-keith
Copy link
Contributor Author

run buildall

@yx-keith
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33640 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72187928ab138155569a9c3c9a145dad333e31f5, data reload: false

------ Round 1 ----------------------------------
q1	26101	5204	5024	5024
q2	1949	270	187	187
q3	10291	1237	702	702
q4	10215	966	518	518
q5	7533	2363	2301	2301
q6	171	160	132	132
q7	893	726	621	621
q8	9313	1292	1000	1000
q9	6942	5074	5028	5028
q10	6844	2316	1902	1902
q11	484	293	280	280
q12	343	357	215	215
q13	17770	3650	3027	3027
q14	225	226	225	225
q15	564	484	504	484
q16	424	425	378	378
q17	581	843	343	343
q18	7613	7144	7146	7144
q19	1528	955	556	556
q20	333	331	211	211
q21	3825	2548	2363	2363
q22	1021	1041	999	999
Total cold run time: 114963 ms
Total hot run time: 33640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5095	5084	5083	5083
q2	234	312	220	220
q3	2120	2622	2249	2249
q4	1322	1775	1379	1379
q5	4314	4352	4395	4352
q6	219	174	127	127
q7	2011	1916	1785	1785
q8	2611	2555	2534	2534
q9	7170	7191	6973	6973
q10	2954	3190	2782	2782
q11	575	522	514	514
q12	689	744	605	605
q13	3534	3912	3290	3290
q14	283	302	278	278
q15	524	494	471	471
q16	452	484	438	438
q17	1176	1500	1421	1421
q18	7775	7419	7451	7419
q19	843	803	871	803
q20	1956	2029	1923	1923
q21	4801	4416	4433	4416
q22	1111	1036	1027	1027
Total cold run time: 51769 ms
Total hot run time: 50089 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193311 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72187928ab138155569a9c3c9a145dad333e31f5, data reload: false

query1	1401	1094	1090	1090
query2	6337	1840	1806	1806
query3	11005	4428	4427	4427
query4	52993	25543	23733	23733
query5	5262	523	465	465
query6	374	214	205	205
query7	4981	502	298	298
query8	299	240	228	228
query9	6308	2639	2643	2639
query10	432	335	266	266
query11	15236	15078	14895	14895
query12	164	112	114	112
query13	1107	532	404	404
query14	10071	6301	6271	6271
query15	211	189	180	180
query16	7037	677	495	495
query17	1076	732	588	588
query18	1505	399	299	299
query19	196	191	160	160
query20	128	122	118	118
query21	201	124	110	110
query22	4424	4506	4196	4196
query23	34387	33996	33857	33857
query24	6649	2386	2441	2386
query25	472	494	411	411
query26	702	272	151	151
query27	2235	509	336	336
query28	3063	2177	2172	2172
query29	590	547	462	462
query30	270	225	183	183
query31	865	839	779	779
query32	74	68	70	68
query33	489	368	344	344
query34	759	888	539	539
query35	793	841	757	757
query36	937	988	915	915
query37	121	106	79	79
query38	4152	4212	4201	4201
query39	1562	1472	1453	1453
query40	208	126	115	115
query41	64	54	55	54
query42	132	113	110	110
query43	494	511	494	494
query44	1339	842	830	830
query45	188	181	163	163
query46	848	1025	659	659
query47	1853	1924	1829	1829
query48	412	435	329	329
query49	656	495	392	392
query50	671	682	412	412
query51	4229	4306	4234	4234
query52	112	107	103	103
query53	218	252	188	188
query54	577	575	529	529
query55	89	82	87	82
query56	301	302	285	285
query57	1234	1247	1162	1162
query58	266	261	270	261
query59	2706	2753	2697	2697
query60	344	340	304	304
query61	125	124	131	124
query62	736	723	684	684
query63	225	192	192	192
query64	1483	1069	741	741
query65	4383	4243	4203	4203
query66	738	396	306	306
query67	16077	15670	15371	15371
query68	7355	902	524	524
query69	529	315	267	267
query70	1217	1144	1104	1104
query71	502	323	302	302
query72	6017	4789	4764	4764
query73	1347	599	362	362
query74	9129	9291	8907	8907
query75	3581	3210	2706	2706
query76	4164	1187	768	768
query77	618	370	281	281
query78	10103	10274	9352	9352
query79	2402	813	569	569
query80	707	510	433	433
query81	491	320	218	218
query82	428	125	94	94
query83	410	242	232	232
query84	298	101	84	84
query85	801	386	322	322
query86	408	313	303	303
query87	4404	4382	4299	4299
query88	3323	2252	2242	2242
query89	399	304	285	285
query90	1823	206	267	206
query91	138	137	112	112
query92	70	57	61	57
query93	2106	924	582	582
query94	662	418	307	307
query95	369	296	290	290
query96	482	565	279	279
query97	2758	2747	2644	2644
query98	236	213	201	201
query99	1411	1367	1284	1284
Total cold run time: 298264 ms
Total hot run time: 193311 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants