Skip to content

[fix](cloud) fix file cache types priority order #51463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 10, 2025

Conversation

freemandealer
Copy link
Contributor

TTL is first-class citizen in Doris FileCache. But the original order make TTL override by INDEX if the context happens to be index of TTL data. We ensure it will be TTL in this PR.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

TTL is first-class citizen in Doris FileCache. But the original
order make TTL override by INDEX if the context happens to be
index of TTL data. We ensure it will be TTL in this PR.

Signed-off-by: zhengyu <[email protected]>
@Thearas
Copy link
Contributor

Thearas commented Jun 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

run buildall

Copy link
Contributor

@kaijchen kaijchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Jun 3, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 33871 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 463c3d8f80d628c2672e93488826ca01717e84bf, data reload: false

------ Round 1 ----------------------------------
q1	26578	5065	5139	5065
q2	1951	278	183	183
q3	10385	1260	697	697
q4	10241	1005	529	529
q5	7625	2387	2338	2338
q6	196	177	130	130
q7	909	736	608	608
q8	9318	1297	1076	1076
q9	6824	5089	5117	5089
q10	6881	2316	1887	1887
q11	478	282	276	276
q12	347	362	222	222
q13	17763	3653	3074	3074
q14	227	237	213	213
q15	573	483	497	483
q16	427	433	367	367
q17	609	873	372	372
q18	7827	7252	7206	7206
q19	2318	989	566	566
q20	328	341	211	211
q21	3674	2602	2325	2325
q22	1047	1003	954	954
Total cold run time: 116526 ms
Total hot run time: 33871 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5246	5064	5112	5064
q2	246	325	227	227
q3	2143	2633	2301	2301
q4	1338	1799	1479	1479
q5	4521	4443	4403	4403
q6	215	164	122	122
q7	1921	1923	1715	1715
q8	2599	2415	2445	2415
q9	7147	7109	7158	7109
q10	2997	3197	2749	2749
q11	576	526	490	490
q12	690	753	614	614
q13	3433	3886	3217	3217
q14	285	288	276	276
q15	520	489	474	474
q16	435	477	424	424
q17	1139	1581	1442	1442
q18	7835	7584	7407	7407
q19	825	809	862	809
q20	2006	2035	1845	1845
q21	4933	4375	4148	4148
q22	1099	1019	983	983
Total cold run time: 52149 ms
Total hot run time: 49713 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185338 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 463c3d8f80d628c2672e93488826ca01717e84bf, data reload: false

query1	1033	500	471	471
query2	6572	1872	1820	1820
query3	6760	233	225	225
query4	26121	23154	23029	23029
query5	4317	622	462	462
query6	309	221	211	211
query7	4625	502	291	291
query8	281	255	219	219
query9	8632	2678	2675	2675
query10	507	351	274	274
query11	15283	15116	14698	14698
query12	157	107	104	104
query13	1663	528	430	430
query14	8740	6035	6113	6035
query15	221	189	171	171
query16	7270	649	442	442
query17	1186	724	570	570
query18	1983	389	296	296
query19	196	181	163	163
query20	121	130	118	118
query21	223	124	112	112
query22	3967	4005	4052	4005
query23	33788	32948	32943	32943
query24	8474	2336	2358	2336
query25	539	465	401	401
query26	1242	266	147	147
query27	2738	495	342	342
query28	4374	2156	2150	2150
query29	799	561	435	435
query30	280	220	185	185
query31	925	829	746	746
query32	75	65	66	65
query33	551	380	304	304
query34	787	845	548	548
query35	794	816	729	729
query36	977	990	875	875
query37	116	97	74	74
query38	4113	4159	4110	4110
query39	1459	1389	1402	1389
query40	218	121	110	110
query41	62	62	61	61
query42	128	112	111	111
query43	541	502	483	483
query44	1309	817	806	806
query45	178	175	169	169
query46	831	1009	625	625
query47	1744	1762	1727	1727
query48	400	431	312	312
query49	741	479	428	428
query50	640	684	399	399
query51	4121	4117	4142	4117
query52	104	109	100	100
query53	223	250	188	188
query54	575	594	517	517
query55	89	83	90	83
query56	319	283	292	283
query57	1134	1146	1118	1118
query58	261	257	254	254
query59	2630	2744	2629	2629
query60	329	311	301	301
query61	140	127	131	127
query62	830	756	687	687
query63	223	193	188	188
query64	4367	1001	711	711
query65	4261	4129	4112	4112
query66	1137	416	298	298
query67	15767	15464	15696	15464
query68	7816	879	527	527
query69	462	301	261	261
query70	1148	1089	1060	1060
query71	460	322	314	314
query72	5571	4707	4807	4707
query73	647	624	346	346
query74	9079	9194	8893	8893
query75	3837	3212	2684	2684
query76	3482	1186	770	770
query77	788	406	283	283
query78	9946	10075	9339	9339
query79	2318	783	578	578
query80	621	513	445	445
query81	482	255	227	227
query82	442	132	96	96
query83	288	250	231	231
query84	290	115	93	93
query85	791	357	323	323
query86	326	301	265	265
query87	4390	4516	4305	4305
query88	3175	2302	2272	2272
query89	376	316	280	280
query90	1926	213	208	208
query91	133	150	115	115
query92	76	63	57	57
query93	1108	939	587	587
query94	677	424	306	306
query95	382	312	295	295
query96	497	563	287	287
query97	2742	2756	2664	2664
query98	233	213	204	204
query99	1471	1421	1308	1308
Total cold run time: 271480 ms
Total hot run time: 185338 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 463c3d8f80d628c2672e93488826ca01717e84bf, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.27	0.19	0.19
query4	1.59	0.18	0.19
query5	0.45	0.41	0.43
query6	1.16	0.67	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.51
query10	0.57	0.57	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.60
query14	0.79	0.80	0.82
query15	0.87	0.86	0.85
query16	0.39	0.38	0.39
query17	1.03	1.03	1.04
query18	0.23	0.21	0.22
query19	1.95	1.82	1.78
query20	0.01	0.02	0.01
query21	15.43	0.95	0.56
query22	0.77	1.34	0.74
query23	14.75	1.39	0.63
query24	7.49	0.92	0.37
query25	0.49	0.25	0.06
query26	0.55	0.16	0.13
query27	0.05	0.04	0.05
query28	8.61	0.97	0.45
query29	12.57	4.01	3.38
query30	0.26	0.10	0.06
query31	2.82	0.62	0.38
query32	3.23	0.55	0.46
query33	3.05	3.07	3.06
query34	15.90	5.11	4.46
query35	4.48	4.54	4.51
query36	0.67	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.05	0.03	0.04
Total cold run time: 102.64 s
Total hot run time: 28.65 s

dataroaring
dataroaring previously approved these changes Jun 4, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Jun 4, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2025
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 4, 2025
Signed-off-by: zhengyu <[email protected]>
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34003 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a264b218da0c62f46d4ae1657a13b02f73caacb, data reload: false

------ Round 1 ----------------------------------
q1	25995	5171	4997	4997
q2	1944	272	182	182
q3	10416	1261	702	702
q4	10224	1006	554	554
q5	7608	2395	2347	2347
q6	179	163	134	134
q7	908	737	629	629
q8	9308	1305	1112	1112
q9	6749	5066	5110	5066
q10	6875	2308	1897	1897
q11	496	288	296	288
q12	342	351	221	221
q13	17774	3720	3149	3149
q14	238	225	218	218
q15	560	480	497	480
q16	426	430	380	380
q17	593	861	355	355
q18	7827	7139	7102	7102
q19	1737	959	590	590
q20	343	339	232	232
q21	3820	3361	2390	2390
q22	1057	1012	978	978
Total cold run time: 115419 ms
Total hot run time: 34003 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5204	5072	5078	5072
q2	235	316	225	225
q3	2183	2670	2315	2315
q4	1348	1765	1361	1361
q5	4542	4370	4449	4370
q6	222	174	130	130
q7	2008	1919	1823	1823
q8	2621	2684	2579	2579
q9	7189	7206	7061	7061
q10	3017	3158	2748	2748
q11	573	506	494	494
q12	700	815	616	616
q13	3512	3854	3373	3373
q14	302	312	289	289
q15	548	501	497	497
q16	451	478	444	444
q17	1185	1570	1396	1396
q18	7716	7656	7509	7509
q19	832	856	923	856
q20	1990	1990	1848	1848
q21	4911	4397	4541	4397
q22	1147	1043	1037	1037
Total cold run time: 52436 ms
Total hot run time: 50440 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193761 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a264b218da0c62f46d4ae1657a13b02f73caacb, data reload: false

query1	1411	1079	1040	1040
query2	6362	1865	1852	1852
query3	11141	4780	4589	4589
query4	25534	23721	23277	23277
query5	4277	629	469	469
query6	312	210	204	204
query7	3993	506	293	293
query8	280	225	217	217
query9	8471	2660	2651	2651
query10	454	338	273	273
query11	15598	15251	15071	15071
query12	168	109	108	108
query13	1558	551	420	420
query14	9093	6271	6131	6131
query15	200	199	186	186
query16	7641	656	525	525
query17	1191	761	645	645
query18	2036	418	313	313
query19	211	193	176	176
query20	123	120	122	120
query21	218	132	114	114
query22	4455	4419	4360	4360
query23	34543	33720	33763	33720
query24	8375	2457	2366	2366
query25	513	467	410	410
query26	741	272	150	150
query27	3082	522	347	347
query28	4354	2244	2193	2193
query29	646	584	448	448
query30	286	220	201	201
query31	892	854	781	781
query32	83	66	63	63
query33	541	361	333	333
query34	848	873	548	548
query35	813	836	774	774
query36	946	991	905	905
query37	110	100	74	74
query38	4297	4490	4229	4229
query39	1543	1429	1437	1429
query40	219	120	106	106
query41	59	58	56	56
query42	127	113	117	113
query43	517	521	507	507
query44	1345	844	839	839
query45	189	176	165	165
query46	867	1031	680	680
query47	1831	1862	1786	1786
query48	399	435	329	329
query49	700	495	405	405
query50	690	701	420	420
query51	4289	4295	4214	4214
query52	107	108	110	108
query53	240	262	182	182
query54	603	586	521	521
query55	92	87	89	87
query56	296	307	291	291
query57	1188	1220	1127	1127
query58	273	260	252	252
query59	2767	2844	2677	2677
query60	347	318	309	309
query61	134	122	132	122
query62	788	743	683	683
query63	231	190	187	187
query64	2983	1061	704	704
query65	4310	4285	4229	4229
query66	871	400	309	309
query67	16126	15783	15740	15740
query68	9401	908	531	531
query69	489	304	268	268
query70	1216	1108	1111	1108
query71	460	311	305	305
query72	5366	4672	4797	4672
query73	688	595	358	358
query74	9201	9178	9035	9035
query75	4367	3206	2681	2681
query76	3662	1172	812	812
query77	934	367	286	286
query78	9947	10127	9347	9347
query79	1847	828	584	584
query80	672	510	454	454
query81	484	251	218	218
query82	252	131	102	102
query83	292	263	232	232
query84	289	111	79	79
query85	764	352	384	352
query86	326	315	299	299
query87	4442	4554	4332	4332
query88	2876	2297	2264	2264
query89	416	310	288	288
query90	2029	209	212	209
query91	155	149	114	114
query92	76	61	63	61
query93	2178	940	584	584
query94	667	408	316	316
query95	386	297	282	282
query96	495	572	289	289
query97	2724	2730	2642	2642
query98	224	207	217	207
query99	1395	1393	1297	1297
Total cold run time: 278648 ms
Total hot run time: 193761 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a264b218da0c62f46d4ae1657a13b02f73caacb, data reload: false

query1	0.03	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.18	0.19
query4	1.59	0.19	0.19
query5	0.45	0.46	0.43
query6	1.53	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.53	0.51
query10	0.59	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.80	0.81	0.80
query15	0.88	0.84	0.86
query16	0.39	0.37	0.38
query17	1.05	1.05	1.00
query18	0.22	0.21	0.21
query19	1.90	1.82	1.79
query20	0.01	0.01	0.02
query21	15.40	0.89	0.56
query22	0.75	1.20	0.71
query23	14.92	1.38	0.59
query24	6.62	1.60	0.33
query25	0.33	0.27	0.06
query26	0.60	0.17	0.15
query27	0.04	0.05	0.05
query28	9.31	0.92	0.44
query29	12.55	4.04	3.31
query30	0.25	0.10	0.07
query31	2.81	0.58	0.39
query32	3.23	0.55	0.47
query33	3.01	3.09	3.06
query34	15.83	5.11	4.48
query35	4.52	4.53	4.59
query36	0.69	0.50	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 102.73 s
Total hot run time: 28.61 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 9, 2025
Copy link
Contributor

github-actions bot commented Jun 9, 2025

PR approved by at least one committer and no changes requested.

@freemandealer
Copy link
Contributor Author

run external

@freemandealer
Copy link
Contributor Author

run beut

@dataroaring dataroaring merged commit 8f4be76 into apache:master Jun 10, 2025
22 of 24 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 10, 2025
TTL is first-class citizen in Doris FileCache. But the original order
make TTL override by INDEX if the context happens to be index of TTL
data. We ensure it will be TTL in this PR.

Signed-off-by: zhengyu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants