Skip to content

[Fix](routine load) Fix routine load partial update #59209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Dec 23, 2025
Merged

[Fix](routine load) Fix routine load partial update #59209

dataroaring merged 3 commits into from
Dec 23, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Dec 19, 2025 •
edited
Loading

What problem does this PR solve?

  1. fix"partial_colunms"="true" doesn't take effect in routine load
  2. add "partial_update_new_key_behavior"="APPEND"/"ERROR" property in routine load

doc: apache/doris-website#3211

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 19, 2025

TPC-H: Total hot run time: 34813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a77e80c985ebdabcaab247f83c81f2f795a9001, data reload: false

------ Round 1 ----------------------------------
q1	17630	4217	4087	4087
q2	2042	363	239	239
q3	10155	1341	731	731
q4	10221	816	315	315
q5	7541	2149	1865	1865
q6	179	168	136	136
q7	1002	846	707	707
q8	9342	1424	1185	1185
q9	6935	5248	5304	5248
q10	6828	2378	1979	1979
q11	496	317	284	284
q12	645	724	599	599
q13	17778	3638	3024	3024
q14	290	302	277	277
q15	601	515	507	507
q16	708	700	636	636
q17	686	760	573	573
q18	7525	7084	7003	7003
q19	1108	960	609	609
q20	398	367	246	246
q21	4226	3975	3619	3619
q22	1009	991	944	944
Total cold run time: 107345 ms
Total hot run time: 34813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4096	4039	4075	4039
q2	322	407	315	315
q3	2125	2696	2311	2311
q4	1356	1753	1273	1273
q5	4205	4454	4715	4454
q6	239	189	139	139
q7	2084	1938	1842	1842
q8	2696	2583	2478	2478
q9	7553	7415	7518	7415
q10	3096	3286	2814	2814
q11	602	513	501	501
q12	693	749	690	690
q13	3536	4079	3194	3194
q14	300	304	270	270
q15	534	498	524	498
q16	643	730	647	647
q17	1208	1616	1445	1445
q18	7869	7604	7493	7493
q19	942	857	882	857
q20	1949	2027	2005	2005
q21	4824	4309	4115	4115
q22	1125	1120	985	985
Total cold run time: 51997 ms
Total hot run time: 49780 ms

@doris-robot
Copy link

doris-robot commented Dec 19, 2025

TPC-DS: Total hot run time: 177908 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7a77e80c985ebdabcaab247f83c81f2f795a9001, data reload: false

query5	4518	622	448	448
query6	350	245	220	220
query7	4224	465	289	289
query8	315	257	262	257
query9	8777	2542	2528	2528
query10	525	381	339	339
query11	15140	14826	14670	14670
query12	183	124	122	122
query13	1284	498	378	378
query14	6192	2966	2693	2693
query14_1	2624	2585	2627	2585
query15	213	201	176	176
query16	850	457	472	457
query17	1120	677	563	563
query18	2474	427	347	347
query19	227	226	198	198
query20	120	116	142	116
query21	220	140	117	117
query22	3910	4032	3912	3912
query23	16539	16315	16091	16091
query23_1	16098	15971	15991	15971
query24	7417	1627	1218	1218
query24_1	1244	1241	1246	1241
query25	562	476	413	413
query26	1250	264	155	155
query27	2793	471	306	306
query28	4465	2122	2117	2117
query29	792	555	433	433
query30	316	247	216	216
query31	817	710	630	630
query32	81	72	69	69
query33	526	343	291	291
query34	894	916	536	536
query35	807	807	718	718
query36	868	903	825	825
query37	129	92	76	76
query38	2879	2819	2797	2797
query39	742	739	709	709
query39_1	721	698	695	695
query40	228	140	121	121
query41	66	63	60	60
query42	106	103	110	103
query43	422	430	394	394
query44	1300	738	742	738
query45	190	187	182	182
query46	870	968	612	612
query47	1667	1705	1608	1608
query48	314	321	246	246
query49	617	426	365	365
query50	652	297	215	215
query51	3821	3845	3885	3845
query52	105	106	98	98
query53	322	348	288	288
query54	283	265	247	247
query55	76	77	73	73
query56	292	292	291	291
query57	1176	1122	1080	1080
query58	269	278	253	253
query59	2436	2571	2331	2331
query60	326	310	301	301
query61	187	194	195	194
query62	726	675	647	647
query63	335	293	309	293
query64	5082	1312	1006	1006
query65	4020	3948	3929	3929
query66	1452	424	319	319
query67	15147	14949	14618	14618
query68	2716	1025	770	770
query69	445	345	303	303
query70	1044	1020	992	992
query71	319	295	270	270
query72	6114	5021	5154	5021
query73	498	565	310	310
query74	8779	8817	8539	8539
query75	3118	3242	2825	2825
query76	2919	1121	726	726
query77	350	397	297	297
query78	9427	9636	8838	8838
query79	1125	872	621	621
query80	756	705	592	592
query81	484	287	247	247
query82	1352	143	108	108
query83	268	260	245	245
query84	264	126	110	110
query85	979	523	465	465
query86	308	280	280	280
query87	3012	3022	3027	3022
query88	3253	2287	2261	2261
query89	456	417	397	397
query90	2024	155	146	146
query91	171	182	145	145
query92	69	66	67	66
query93	969	894	569	569
query94	365	309	267	267
query95	559	326	300	300
query96	611	467	208	208
query97	2269	2305	2204	2204
query98	226	197	199	197
query99	1299	1283	1217	1217
Total cold run time: 251992 ms
Total hot run time: 177908 ms

@doris-robot
Copy link

doris-robot commented Dec 19, 2025

ClickBench: Total hot run time: 27.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a77e80c985ebdabcaab247f83c81f2f795a9001, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.26	0.26	0.25
query6	1.15	0.64	0.63
query7	0.04	0.03	0.02
query8	0.06	0.04	0.04
query9	0.57	0.52	0.52
query10	0.56	0.56	0.57
query11	0.15	0.11	0.12
query12	0.16	0.11	0.12
query13	0.62	0.60	0.60
query14	1.00	0.99	0.99
query15	0.82	0.80	0.81
query16	0.40	0.40	0.42
query17	1.03	1.00	1.05
query18	0.23	0.21	0.21
query19	1.91	1.85	1.79
query20	0.02	0.01	0.02
query21	15.44	0.29	0.14
query22	4.99	0.05	0.05
query23	16.12	0.30	0.11
query24	1.37	0.25	0.46
query25	0.07	0.05	0.07
query26	0.14	0.13	0.14
query27	0.12	0.06	0.05
query28	4.14	1.22	1.02
query29	12.55	3.95	3.18
query30	0.28	0.13	0.12
query31	2.82	0.63	0.41
query32	3.22	0.55	0.46
query33	2.96	3.10	3.02
query34	16.82	5.11	4.55
query35	4.66	4.58	4.56
query36	0.67	0.50	0.51
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.13	0.13
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.96 s
Total hot run time: 27.27 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 19, 2025

FE Regression Coverage Report

Increment line coverage 82.93% (34/41) 🎉
Increment coverage report
Complete coverage report

@dataroaring
Copy link
Contributor

dataroaring commented Dec 19, 2025

Please add a document pr.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 22, 2025

run p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 22, 2025

run external

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 22, 2025

run vault_p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 22, 2025

run external

liaoxin01
liaoxin01 approved these changes Dec 23, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 23, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 23, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Dec 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 670deb9 into apache:master Dec 23, 2025
34 of 35 checks passed
bobhan1 added a commit to bobhan1/doris that referenced this pull request Dec 24, 2025
@bobhan1
[Fix](routine load) Fix routine load partial update (apache#59209)
3c87e20 
1. fix`"partial_colunms"="true"` doesn't take effect in routine load
2. add `"partial_update_new_key_behavior"="APPEND"/"ERROR"` property in
routine load

doc: apache/doris-website#3211
@bobhan1 bobhan1 mentioned this pull request Dec 24, 2025
Merged
bobhan1 added a commit to bobhan1/doris that referenced this pull request Dec 24, 2025
@bobhan1
[Fix](routine load) Fix routine load partial update (apache#59209)
1f58deb 
1. fix`"partial_colunms"="true"` doesn't take effect in routine load
2. add `"partial_update_new_key_behavior"="APPEND"/"ERROR"` property in
routine load

doc: apache/doris-website#3211
@bobhan1 bobhan1 mentioned this pull request Dec 24, 2025
Merged
yiguolei pushed a commit that referenced this pull request Dec 24, 2025
@bobhan1
branch-4.0: [Fix](routine load) Fix routine load partial update (#59209…
144bd4b 
…) (#59308)

pick #59209
morrySnow pushed a commit that referenced this pull request Dec 25, 2025
@bobhan1
branch-3.1: [Fix](routine load) Fix routine load partial update #59209 (
79a027e 
#59313)

pick #59209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bobhan1 @Thearas @doris-robot @hello-stephen @dataroaring @liaoxin01 @yiguolei @morrySnow