Skip to content

[opt](varbinary) support multi_distinct_count about varbinary and add conf mapping iceberg uuid #59406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Dec 29, 2025
Merged

[opt](varbinary) support multi_distinct_count about varbinary and add conf mapping iceberg uuid #59406

morningman merged 1 commit into from
Dec 29, 2025
+17 −8

Conversation

@zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented Dec 26, 2025 •
edited
Loading

What problem does this PR solve?

doc apache/doris-website#3231

  1. iceberg uuid type use the conf of enable.varbinary.mapping to control mapping type, default is mapping to string type.
  2. as binary type not support used in group by key, so count(distinct binary) will thrown error now, support binary type in multi_distinct_count agg function.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333
Copy link
Contributor Author

zhangstar333 commented Dec 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 26, 2025

TPC-H: Total hot run time: 36054 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 918a4e9a3d2e3e34d192543dec1fb26357610070, data reload: false

------ Round 1 ----------------------------------
q1	17646	4290	4072	4072
q2	2028	358	236	236
q3	10170	1319	764	764
q4	10237	897	321	321
q5	7519	2083	1990	1990
q6	192	165	137	137
q7	1008	843	717	717
q8	9363	1430	1263	1263
q9	6993	5312	5384	5312
q10	6790	2421	1990	1990
q11	528	332	306	306
q12	667	744	564	564
q13	17807	3699	3069	3069
q14	286	293	274	274
q15	580	513	516	513
q16	689	689	635	635
q17	699	892	433	433
q18	7257	7374	7939	7374
q19	950	993	638	638
q20	466	411	265	265
q21	4591	4312	4142	4142
q22	1175	1085	1039	1039
Total cold run time: 107641 ms
Total hot run time: 36054 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4262	4210	4210
q2	331	406	325	325
q3	2417	2933	2492	2492
q4	1450	1916	1436	1436
q5	4741	4533	4566	4533
q6	216	170	127	127
q7	2054	1945	1807	1807
q8	2649	2567	2525	2525
q9	7545	7529	7508	7508
q10	3083	3272	2799	2799
q11	614	551	528	528
q12	671	766	569	569
q13	3292	3654	3116	3116
q14	277	291	251	251
q15	532	494	491	491
q16	660	661	615	615
q17	1130	1385	1462	1385
q18	7379	7206	7009	7009
q19	853	810	847	810
q20	1919	2013	1787	1787
q21	4664	4330	4217	4217
q22	1096	1018	1005	1005
Total cold run time: 51904 ms
Total hot run time: 49545 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2025

TPC-DS: Total hot run time: 179643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 918a4e9a3d2e3e34d192543dec1fb26357610070, data reload: false

query5	5400	610	453	453
query6	330	254	223	223
query7	4237	464	275	275
query8	328	267	251	251
query9	8778	2541	2582	2541
query10	553	388	332	332
query11	15573	14894	14856	14856
query12	208	118	121	118
query13	1267	512	384	384
query14	6572	2990	2756	2756
query14_1	2722	2658	2631	2631
query15	201	204	183	183
query16	872	474	447	447
query17	1133	728	612	612
query18	2721	456	354	354
query19	238	233	205	205
query20	124	116	115	115
query21	226	150	118	118
query22	4095	3966	3973	3966
query23	16721	16203	15996	15996
query23_1	16300	16229	16062	16062
query24	7271	1655	1228	1228
query24_1	1229	1246	1267	1246
query25	564	482	428	428
query26	1248	267	153	153
query27	2764	461	303	303
query28	4433	2149	2147	2147
query29	777	570	447	447
query30	315	245	218	218
query31	835	686	638	638
query32	77	70	67	67
query33	574	342	292	292
query34	896	912	542	542
query35	757	796	706	706
query36	863	887	824	824
query37	132	98	78	78
query38	3021	3091	2987	2987
query39	818	740	723	723
query39_1	705	708	731	708
query40	226	138	126	126
query41	67	64	65	64
query42	110	112	110	110
query43	431	428	401	401
query44	1359	745	741	741
query45	192	195	188	188
query46	883	982	608	608
query47	1661	1691	1654	1654
query48	317	325	251	251
query49	636	438	359	359
query50	663	289	228	228
query51	3778	3835	3909	3835
query52	105	109	100	100
query53	315	355	296	296
query54	291	260	256	256
query55	80	81	76	76
query56	310	300	303	300
query57	1145	1156	1104	1104
query58	275	254	249	249
query59	2417	2530	2359	2359
query60	322	317	302	302
query61	172	164	166	164
query62	741	712	695	695
query63	323	302	294	294
query64	4934	1328	1027	1027
query65	4028	3974	3942	3942
query66	1384	438	333	333
query67	15210	14940	15197	14940
query68	3459	1050	737	737
query69	509	361	365	361
query70	1049	963	1001	963
query71	379	305	295	295
query72	6100	4981	4952	4952
query73	537	567	307	307
query74	8912	9008	8668	8668
query75	3189	3156	2792	2792
query76	3544	1134	723	723
query77	544	415	280	280
query78	9359	9464	8853	8853
query79	1184	964	609	609
query80	1579	667	545	545
query81	570	269	237	237
query82	560	136	105	105
query83	376	251	240	240
query84	265	126	104	104
query85	966	519	476	476
query86	427	300	281	281
query87	3212	3252	3092	3092
query88	3328	2301	2289	2289
query89	463	423	389	389
query90	1915	161	154	154
query91	181	168	149	149
query92	75	65	71	65
query93	1087	943	552	552
query94	567	313	278	278
query95	570	324	306	306
query96	614	484	210	210
query97	2314	2306	2224	2224
query98	206	203	210	203
query99	1316	1333	1266	1266
Total cold run time: 257844 ms
Total hot run time: 179643 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2025

ClickBench: Total hot run time: 27.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 918a4e9a3d2e3e34d192543dec1fb26357610070, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.05	0.06
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.28	0.28	0.26
query6	1.18	0.64	0.63
query7	0.04	0.03	0.03
query8	0.06	0.05	0.04
query9	0.57	0.51	0.51
query10	0.55	0.55	0.54
query11	0.15	0.10	0.12
query12	0.15	0.11	0.12
query13	0.62	0.59	0.59
query14	0.98	0.98	0.97
query15	0.82	0.79	0.80
query16	0.38	0.38	0.43
query17	0.96	1.05	1.04
query18	0.23	0.22	0.21
query19	1.87	1.79	1.77
query20	0.03	0.01	0.01
query21	15.45	0.28	0.15
query22	4.93	0.05	0.04
query23	15.98	0.29	0.10
query24	1.54	0.62	0.74
query25	0.08	0.09	0.05
query26	0.15	0.14	0.14
query27	0.06	0.06	0.08
query28	5.01	1.20	1.02
query29	12.63	3.95	3.27
query30	0.28	0.13	0.11
query31	2.83	0.64	0.39
query32	3.23	0.54	0.45
query33	2.96	3.02	2.96
query34	16.83	5.19	4.53
query35	4.60	4.56	4.62
query36	0.66	0.49	0.48
query37	0.10	0.07	0.07
query38	0.08	0.05	0.03
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 98.68 s
Total hot run time: 27.56 s

@doris-robot
Copy link

doris-robot commented Dec 26, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.40% (18952/35489)
Line Coverage 39.26% (175762/447674)
Region Coverage 33.82% (135947/402007)
Branch Coverage 34.75% (58712/168938)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 26, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 26, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.17% (25038/34694)
Line Coverage 58.86% (262813/446482)
Region Coverage 53.68% (218044/406172)
Branch Coverage 55.21% (93577/169502)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 26, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

morningman
morningman approved these changes Dec 28, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 28, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit b1303c3 into apache:master Dec 29, 2025
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 30, 2025
@zhangstar333
[opt](varbinary) support multi_distinct_count about varbinary and add…
ad4bf10 
… conf mapping iceberg uuid (#59406)

### What problem does this PR solve?

doc apache/doris-website#3231

1. iceberg uuid type use the conf of enable.varbinary.mapping to control
mapping type, default is mapping to string type.
2. as binary type not support used in group by key, so count(distinct
binary) will thrown error now, support binary type in
multi_distinct_count agg function.
@github-actions github-actions bot mentioned this pull request Dec 30, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.x reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zhangstar333 @hello-stephen @doris-robot @morningman @CalvinKirs