Skip to content

[opt](expr) Preserve eligible simple CASE expressions for direct key matching - #69296

Open
Mryange wants to merge 7 commits into
apache:masterfrom
Mryange:opt-simple-case-evaluation
Open

Mryange wants to merge 7 commits into
apache:masterfrom
Mryange:opt-simple-case-evaluation

Conversation

@Mryange

@Mryange Mryange commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Problem Summary:

Nereids translates simple CASE expressions into searched CASE, materializing a separate equality result for each WHEN key. This PR adds two optimizations behind enable_experimental_simple_case (disabled by default):

  • Preserve eligible expressions as simple CASE, evaluate their common input once, and compare it directly with literal WHEN keys in BE. Numeric matching loops allow automatic vectorization.
  • For sufficiently many WHEN branches, build a fragment-local hash table from literal keys to the first matching branch. Each input row then performs one lookup instead of comparing against every key. Small expressions and constant inputs retain linear matching. Constant THEN columns are read directly without expanding them into full columns.

Preserve first-match results and batch-level THEN evaluation, including duplicate keys. FE eligibility remains restricted to compatible integer, DecimalV3 and string comparisons with non-NULL literal keys; unsupported expression shapes and short-circuit evaluation retain searched CASE. Remove the former 256-branch cap in favor of the BE branch-index capacity; the general FE expression-size limit still applies.

Real SQL measurements comparing lookup with linear matching, with the same constant-THEN optimization in both variants, show benefits for many branches. For 256 branches with mixed hits and misses, measured SQL speedups were 1.23x for INT, 1.97x for BIGINT and 4.61x for STRING. These are shared-machine results from an earlier implementation: machine load varied, some smaller cases had higher SQL times, and the final type, duplicate-key and allocator changes have not been rebenchmarked.

Enable the experimental setting only after all BEs have been upgraded: older BEs do not support this expression layout.

Release note

Add opt-in simple CASE execution with direct literal matching and hash lookup for expressions with many WHEN branches. The setting is disabled by default and applies only when short-circuit evaluation is disabled.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (details below)
    • No need to test or manual test.

    An earlier implementation passed 104 BE unit tests, 11 FE unit tests, five related regression suites, RELEASE BE/FE builds, and targeted clang-tidy/checkstyle checks. Regression output was generated by the harness and rerun normally. Lookup-versus-linear SQL validation covered 45 workloads over four rounds (1,800 measured queries), with matching results and normalized plans, one scanner, one pipeline instance, default batch size 8160 and caches disabled. Numeric matching and result-merge SIMD was confirmed in compiler records and the built binary; hash probing itself does not auto-vectorize.

    The final numeric-type extension, duplicate-key handling, helper extraction and Doris allocator changes have only formatting, build hygiene and diff checks; they have not been rebuilt or retested.

  • Behavior changed:

    • No.
    • Yes. Eligible CASE expressions use the new BE path only when explicitly enabled; default behavior is unchanged.
  • Does this need documentation?

    • No. Experimental setting, disabled by default.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Mryange
Mryange requested a review from morrySnow as a code owner October 19, 2026 15:34
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all nine changed paths at head 448cb83. No substantiated new findings and no existing blocking inline comments. The review converged in round 1: both complete-scope reviewers and the separate boundary-risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the same empty inline set; all candidates and risk focuses have evidence-backed dispositions.

The traced flow is Nereids parser/analyzer and coercion -> shared-input recognition -> CaseExpr/Thrift layout -> BE factory, prepare/open/clone -> ColumnView matching -> existing result assembly. The type/layout and NULL/constant/selector/skipped-branch risks were dismissed after checking their upstream and downstream guarantees, including session guards and prepared point-query serialization.

Critical checkpoint conclusions:

  1. Goal and evidence: eligible opt-in CASE expressions evaluate the common input once and compare literal keys directly. Added FE/BE/regression tests cover the intended behavior; they were inspected, not executed here.
  2. Scope and reuse: the changes stay within CASE lowering/execution, session gating and tests, reusing ColumnView and result assembly.
  3. Concurrency and locking: no new threads or locks; new execution state is call-local and layout flags are immutable after construction.
  4. Lifecycle and initialization: ColumnPtr owners outlive views; output columns preserve source COW ownership; cloning preserves the layout flag. No new static initialization dependency or ownership cycle.
  5. Configuration: the experimental session flag defaults false, is dynamically read during translation, and uses reflective needForward forwarding. No restart is required for session changes.
  6. Compatibility: existing CASE Thrift fields carry the new layout. Enable the feature only after all BEs support it, as documented; older BEs cannot execute this layout. Default-off operation retains searched CASE. No storage, linker or libc ABI change.
  7. Protocol definitions: no thrift/protobuf definitions changed; field-ID compatibility comparison is not applicable.
  8. Parallel paths: searched CASE, short-circuit evaluation, missing context and unsupported shapes retain their existing layout. Cloud/shared-nothing use the common expression path; constant-folding and point-query sends were checked.
  9. Conditional semantics: exact numeric/decimal types and STRING/VARCHAR aliases match BE dispatch. NULL keys are excluded; NULL payloads cannot match. First match, eager matching-THEN errors, skipped unmatched branches, empty batches and selector-relative rows are preserved.
  10. Test coverage: integer widths, DecimalV3 widths/scales, strings, NULL/constant inputs, duplicate keys, branch errors, selectors, complex results and 256/257 boundaries have coverage. Production prepare/open/deep-clone paths were traced statically where direct unit tests bypass them.
  11. Expected results: all 272 checked-in blocks were checked against the 68 SQL cases and agree across all four setting combinations. No discrepancy found; regression ordering and setup follow repository conventions.
  12. Observability and errors: child Status errors propagate, VExpr validates output size/type, exception boundaries remain present, and diagnostics identify the layout/type. No additional metric is needed for this local expression path.
  13. Transactions and persistence: no transaction, visible-version, delete-bitmap, journal, rowset or failover change. The new session field uses existing persistence support.
  14. Data writes and atomicity: execution constructs fresh result columns and introduces no persistent data-write path.
  15. FE-BE state: existing has_case_expr and expression children convey the selection; no new BE query option or protocol field is required.
  16. Performance and memory: bounded O(rows * branches), at most 256 branches, one common input evaluation and a tracked DorisVector selection buffer. Equality-column materialization is removed; runtime speedup was not measured.
  17. Other checks: BE header layering/budgets/PCH, extern-template pairing and unity-skip coverage passed via the pure-text build-hygiene script. No unrelated change or additional substantiated issue found.

No additional user-provided review focus was supplied. Coverage limits: this review is static-only, as required. No builds, unit/regression execution, sanitizers, runtime rolling-upgrade exercise or benchmarks were run. The checked-in outputs do not validate the final matching loop at runtime, and the PR description reports that final-loop runtime rerun and added unit-test execution remain outstanding. No unresolved code candidate or uncovered changed path remains.

Review converged after 1 round(s).

@Mryange

Mryange commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28299 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 448cb839836ceeeb360e897ce3eaf787d62383d3, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17630	5144	5058	5058
q2	2116	309	257	257
q3	10237	1343	824	824
q4	4677	456	327	327
q5	7426	789	536	536
q6	179	185	155	155
q7	731	822	602	602
q8	9268	1323	1009	1009
q9	5376	4164	4157	4157
q10	6856	1290	1023	1023
q11	453	249	240	240
q12	647	398	286	286
q13	18080	3535	3016	3016
q14	276	268	240	240
q15	q16	733	721	658	658
q17	1203	745	582	582
q18	6995	6155	6104	6104
q19	1102	948	597	597
q20	384	331	223	223
q21	4839	2151	2286	2151
q22	339	294	254	254
Total cold run time: 99547 ms
Total hot run time: 28299 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5611	5606	5606	5606
q2	659	576	534	534
q3	5179	5344	4966	4966
q4	2021	2124	1524	1524
q5	5253	5153	5168	5153
q6	322	251	151	151
q7	2403	2039	1818	1818
q8	3062	2700	2684	2684
q9	8089	7923	7923	7923
q10	3714	3677	3243	3243
q11	630	426	400	400
q12	687	695	659	659
q13	3615	3873	3291	3291
q14	312	306	282	282
q15	q16	725	734	673	673
q17	8225	7464	6856	6856
q18	13052	12174	12878	12174
q19	916	785	789	785
q20	2133	2127	1922	1922
q21	5575	4708	4681	4681
q22	491	447	401	401
Total cold run time: 72674 ms
Total hot run time: 65726 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151035 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 448cb839836ceeeb360e897ce3eaf787d62383d3, data reload: false

query5	4324	572	442	442
query6	419	205	188	188
query7	4820	447	230	230
query8	325	162	142	142
query9	8775	3057	3121	3057
query10	456	322	250	250
query11	6020	3840	3572	3572
query12	148	89	85	85
query13	1269	439	347	347
query14	6151	3889	3642	3642
query14_1	3408	3390	3375	3375
query15	205	191	179	179
query16	926	428	392	392
query17	842	652	555	555
query18	2418	400	304	304
query19	189	153	129	129
query20	83	84	81	81
query21	209	127	109	109
query22	13002	12933	12932	12932
query23	13635	12856	12537	12537
query23_1	12522	12474	12458	12458
query24	7395	808	458	458
query24_1	463	464	464	464
query25	506	391	331	331
query26	1241	252	138	138
query27	2772	438	273	273
query28	4584	1923	1920	1920
query29	1567	587	447	447
query30	301	226	182	182
query31	1013	925	799	799
query32	138	86	94	86
query33	523	303	241	241
query34	887	850	501	501
query35	747	758	656	656
query36	817	819	712	712
query37	123	97	82	82
query38	1775	1764	1699	1699
query39	789	807	738	738
query39_1	744	744	739	739
query40	225	116	97	97
query41	71	69	70	69
query42	91	87	83	83
query43	347	363	315	315
query44	1389	683	703	683
query45	179	190	164	164
query46	842	971	580	580
query47	2850	2899	2954	2899
query48	348	322	241	241
query49	687	402	298	298
query50	686	279	206	206
query51	10444	10458	10734	10458
query52	83	90	72	72
query53	197	211	166	166
query54	250	223	201	201
query55	71	67	73	67
query56	249	239	225	225
query57	1657	1629	1588	1588
query58	294	266	283	266
query59	2126	2081	1968	1968
query60	296	252	244	244
query61	165	165	168	165
query62	389	333	290	290
query63	196	160	162	160
query64	2881	1050	921	921
query65	3312	3255	3275	3255
query66	1783	422	308	308
query67	20122	20140	19949	19949
query68	2953	999	607	607
query69	397	289	236	236
query70	815	760	738	738
query71	298	228	209	209
query72	2603	2608	2221	2221
query73	514	527	303	303
query74	4646	4527	4351	4351
query75	2301	2305	1899	1899
query76	2323	1009	602	602
query77	331	381	263	263
query78	9812	9752	9082	9082
query79	983	865	522	522
query80	1239	425	344	344
query81	596	323	273	273
query82	570	133	103	103
query83	289	195	173	173
query84	323	123	106	106
query85	1081	437	386	386
query86	373	250	234	234
query87	2000	1939	1799	1799
query88	3725	2692	2650	2650
query89	331	282	240	240
query90	1718	163	172	163
query91	163	156	131	131
query92	102	87	93	87
query93	1001	1003	581	581
query94	554	311	277	277
query95	579	358	299	299
query96	670	477	233	233
query97	2471	2432	2355	2355
query98	158	148	148	148
query99	768	766	637	637
Total cold run time: 235162 ms
Total hot run time: 151035 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 448cb839836ceeeb360e897ce3eaf787d62383d3, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.27	0.09	0.08
query4	1.61	0.10	0.10
query5	0.23	0.24	0.24
query6	1.17	0.68	0.69
query7	0.04	0.00	0.00
query8	0.05	0.04	0.04
query9	0.42	0.36	0.37
query10	0.57	0.60	0.58
query11	0.22	0.15	0.14
query12	0.18	0.15	0.14
query13	0.50	0.49	0.49
query14	1.03	1.01	1.02
query15	0.64	0.61	0.61
query16	0.36	0.35	0.35
query17	1.13	1.14	1.12
query18	0.22	0.19	0.19
query19	2.15	2.02	1.96
query20	0.02	0.02	0.01
query21	15.49	0.20	0.14
query22	4.87	0.05	0.05
query23	16.28	0.31	0.12
query24	3.09	0.63	0.51
query25	0.14	0.07	0.05
query26	1.36	0.21	0.15
query27	0.05	0.04	0.05
query28	3.80	0.77	0.38
query29	12.47	4.07	3.23
query30	0.28	0.14	0.10
query31	2.80	0.55	0.32
query32	3.21	0.60	0.51
query33	3.16	3.19	3.15
query34	15.43	4.03	3.41
query35	3.37	3.35	3.38
query36	0.55	0.44	0.42
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.77 s
Total hot run time: 24.43 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 82.35% (28/34) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 61.76% (21/34) 🎉
Increment coverage report
Complete coverage report

@Mryange

Mryange commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

/review

@Mryange
Mryange force-pushed the opt-simple-case-evaluation branch from f120e0b to da848be Compare October 22, 2026 08:23

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #69296 at da848be covered all 11 changed paths and the FE translation → existing TCaseExpr layout → BE prepare/open/clone/execute/result flow. One P2 finding: the new typed lookup test builds its expected values from a different key sequence than the CASE it executes. No production correctness defect was substantiated. No existing inline findings require carry-forward; there was no additional user focus.

Critical checkpoint conclusions:

  • Goal: eligible CASE expressions evaluate one common input and use linear matching or a fragment lookup; the added FE, BE and SQL tests cover this behavior, but the typed-test oracle needs correction.
  • Scope and clarity: changes are focused on CASE translation/execution and tests; constant-result broadcasting also deliberately affects searched CASE.
  • Concurrency and locking: no new threads or locks; fragment lookup state is immutable after open, context clones share ownership, and selections/results are local to execution.
  • Lifecycle and initialization: child literals own borrowed string-key storage; cloned contexts retain the expression tree and shared fragment state. No new static initialization dependency or ownership cycle was found.
  • Configuration: enable_experimental_simple_case defaults to false, is observed during translation, and needForward uses the existing reflective session forwarding path; no restart is needed for new plans.
  • Compatibility: no storage-format or function-symbol change. The simple CASE expression layout requires upgraded BEs; the disabled default and explicit operator opt-in are the stated rolling-upgrade constraint. Short-circuit evaluation retains searched layout.
  • Thrift/protobuf IDs: no gensrc definitions or IDs changed.
  • Parallel paths: disabled/missing context, unsupported types/shapes/coercion, short-circuit, small/constant inputs, and large lookup cases were traced; shared result merging was checked for both searched and simple CASE.
  • Conditions and errors: eligibility and threshold branches have concrete constraints/comments; NULL keys are rejected, first-match selection is preserved, matching duplicate THENs still evaluate the entire batch, unmatched THENs are skipped, and child errors propagate.
  • Test coverage: tests include NULL payloads, constants, empty batches, native widths, DecimalV3 widths/scales, string keys, duplicate keys, nullable/complex results, selectors, cloned contexts, wide indices, threshold boundaries and branch errors. The inline finding explains why a correct implementation fails the newly added typed suite.
  • Expected results: all 39,300 lookup golden rows match an independent fixture calculation. All 272 decimal output blocks (1,604 rows) match independent fixture calculations and agree across the 68 corresponding mode groups; no golden-output error was found.
  • Observability: existing expression errors/type checks and the enhanced debug string are sufficient for this local execution optimization; no distributed operation or new metric requirement was identified.
  • Transactions and persistence: unchanged; no EditLog/replay, visible-version, delete-bitmap or failover path is involved.
  • Writes and atomicity: no data-write or transaction-boundary change is introduced.
  • FE/BE propagation: existing TCaseExpr flags and child ordering carry the layout, and BE cloning preserves has_case_expr; no new query-option field needs parallel send-path updates.
  • Performance and memory: lookup avoids per-key condition columns; constants avoid full-column expansion. Thresholds retain linear matching for small cases. Lookup storage and matching buffers use Doris allocators; batch result merging retains its per-populated-branch scans. Performance claims were not benchmarked during review.
  • Further sweep: decimal/hash equality for eligible inputs, const/nullable storage, NaN/infinity/signed-zero result stores, uint16 branch limits and string lifetimes were rechecked; no additional verified issue remains.

Validation limit: this invocation performed static analysis and review-only fixture calculations. Builds, Doris unit/regression execution and source edits were prohibited and were not performed; the PR description also states its final implementation changes have not been rebuilt or retested. Golden-file generation provenance was not independently verified from the static artifacts.

Review converged after 2 round(s).

Comment thread be/test/exprs/vcase_expr_test.cpp
@HappenLee

HappenLee commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed the latest head, 81992a12f42db6ad01cdebd7ab5edbcd2161ebdf. I have one confirmed test defect and two performance/validation points to address before merging:

  1. [P2] The typed lookup test's expected values do not match its actual WHEN keys. I independently confirmed the existing review finding, which still applies to this head. With TYPE_INT and 63 branches, the oracle assumes keys 0..62, but CASE construction replaces the last key with duplicate 1. Non-NULL row 62 therefore expects 63, although the correct CASE result is ELSE -1. The mismatch also occurs at larger branch counts. Please derive expected results from the actual inserted key sequence, preserving first-match semantics, NULLs and misses. This currently makes correct linear and lookup implementations fail the new test.

  2. The numeric result merge still scans the batch once per evaluated result column. _execute_simple_case_lookup() reduces key matching to one hash lookup per non-NULL row, but update_result_auto_simd() subsequently loops over each populated THEN/ELSE column and all rows. For non-nullable numeric results, merging remains approximately O(rows * evaluated_result_columns); the complete CASE evaluation can therefore still take O(rows * branches) work when many branches are populated. Please benchmark the final implementation with concentrated and distributed hits, misses, constant and dynamic THEN expressions, and branch counts around the lookup thresholds. Measuring a per-row gather against the existing SIMD merge would help establish whether result assembly becomes the next bottleneck. The reported early lookup-versus-linear speedups do not establish the final implementation's end-to-end benefit.

  3. Please validate the final source version, including the default searched CASE path. The PR description states that the final type, duplicate-key and allocator changes have not been rebuilt or retested; the latest output-compaction commit also says it ran against the older 448cb839 service. Please rebuild and run the corrected BE tests, relevant FE tests and regressions against the final implementation. Cover both settings of enable_experimental_simple_case: the constant-THEN broadcasting change in the shared result-merging helper also affects searched CASE when the experimental setting is disabled.

My verification here was static code tracing and independent fixture calculations; I did not build or execute the final Doris implementation.

Comment thread be/src/exprs/vcase_expr.cpp Outdated
@Mryange

Mryange commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #69296 at head b493c2b83230ba74ebdd2361c5022b00d7d1bed7. No new verified findings. All 11 changed files and their relevant upstream/downstream paths were covered. Review converged after 2 rounds, including a final pass over this unchanged review payload.

Existing review context: the wide-decimal hash concern is addressed by the decimal HashCRC32 selection and repeated-limb cases. The duplicate-key test oracle now derives the first match from the actual remapped keys. The native-128 hash follow-up remains known infrastructure context; those inputs are excluded by current FE eligibility. No existing P0/P1 finding was independently confirmed as applicable, and no duplicate comments are added. The focus file supplied no additional concerns.

Critical checkpoint conclusions:

  1. Goal and evidence: the opt-in lowering reconstructs a common integral, DecimalV3, or string input, evaluates it once, and uses linear/hash matching. FE/BE unit sources and dedicated regression/explain cases cover activation and result semantics; execution was not performed.
  2. Scope and reuse: changes are focused on CASE translation/evaluation, its session flag, and tests. They reuse ColumnView, FunctionContext, Doris hashes/allocators, and existing result assembly.
  3. Concurrency and locks: fragment state is initialized before worker context cloning and read immutably during execution; selection buffers are per call. No new threads, locks, or lock-order dependency were introduced.
  4. Lifecycle and initialization: cloned contexts retain the shared expression root and fragment-state owner, preserving borrowed literal string bytes. Independent expression clones rebuild state. RAII handles partial initialization; no new cross-translation-unit static dependency was found.
  5. Configuration: enable_experimental_simple_case defaults to false, uses normal session forwarding/persistence, and is consulted during plan translation without a restart. Short-circuit evaluation takes precedence.
  6. Compatibility: existing TCaseExpr fields are reused; no storage-format change. The documented prerequisite is to upgrade every BE before enabling the flag. Enabled layouts are not supported on old BEs; capability negotiation is not added.
  7. Thrift/protobuf IDs: no gensrc definitions changed; cross-branch field-ID comparison is not applicable.
  8. Parallel paths: searched CASE and short-circuit fallback retain their layouts; linear/hash matching and shared result assembly were checked. Distributed plans, point queries, and constant folding use the common serializer; constant CASE folding cannot select this slot-dependent lowering.
  9. Conditions and errors: eligibility preserves numeric/decimal types and scale, input identity, and non-NULL literal keys. Branch limits, empty/NULL/constant inputs, selectors, first-match results, shadowed duplicate THEN evaluation, and child error propagation were checked against the surrounding APIs.
  10. Test coverage: inspected threshold boundaries, 256/257 and maximum branch counts, nullable payloads, constant/full/mixed columns, cloned contexts, unmatched and failing branches, string edge keys, decimal widths, complex results, coercion fallbacks, and flag combinations. Dedicated single-CASE explain checks account for CSE potentially bypassing lowering in multi-CASE projections.
  11. New expected results: independently derived and matched all 150 lookup count/checksum sections and all 272 decimal output sections (68 canonical cases). This was a static arithmetic/string oracle, not Doris test execution.
  12. Observability: existing CASE rendering distinguishes the selected layout, debug output includes its flags, and Status/type/row-count checks cover failures. No additional production metric need was identified for this local expression change.
  13. Transactions and persistence: transaction logs, committed-data visibility, partition versions, delete bitmaps, and failover behavior are unchanged. The new session field follows existing annotated serialization.
  14. Data writes and atomicity: production write paths are unchanged; no new transaction/atomicity interaction was identified.
  15. FE/BE variable propagation: the flag is an FE plan choice; existing has_case_expr and child layout carry the result through the common serializer, with no new runtime option requiring separate send-path updates.
  16. Performance and memory: lookup tables and matching buffers use Doris allocator paths, and constant numeric THEN columns avoid full materialization. Hash matching has expected linear batch cost; numeric result assembly retains its existing branch-by-row cost. No measured speedup or runtime memory-limit result is claimed.
  17. Other checks: reviewed includes, branch-index widths, dispatch agreement, clone/close behavior, and result nullability normalization. No additional substantiated issue or unresolved concrete candidate remains.

Limits: static-only review as required. No builds, Doris tests/SQL, sanitizers, release-artifact ABI checks, performance benchmarks, or mixed-version cluster runs were performed. The historical generation of the .out files cannot be proved from static artifacts. No changed-file coverage gap remains within that scope.

Review converged after 2 round(s).

@HappenLee

Copy link
Copy Markdown
Contributor

run buildall

HappenLee
HappenLee previously approved these changes Oct 26, 2026

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head b493c2b83230ba74ebdd2361c5022b00d7d1bed7. No additional blocking correctness issue found in the FE translation, BE matching/result assembly, or context lifetime/clone paths. The wide-decimal hash selection and duplicate-key oracle findings are addressed in this head.

I also extracted the actual shared numeric result-assembly function and checked it against per-row selection under UBSan for full/constant/mixed columns, vector tails, NaN/Inf/signed-zero, and uint8/uint16 branch-index boundaries; that isolated probe passed. This is not a substitute for Doris unit/regression execution. I have triggered run buildall for this exact head; merge must wait for the required latest-head checks.

The remaining per-populated-result-column scan cost is an existing result-assembly strategy and a performance follow-up, not a newly demonstrated correctness regression. No final-head SQL speedup or complete Doris test success is claimed here. Native-128 IN/NOT IN hashing is outside this PR's SQL activation scope.

Review checkpoint conclusions
  1. Goal: eligible CASE inputs are evaluated once with linear/hash matching; first-match selection is preserved. The previously reported hash and test-oracle defects are fixed.
  2. Scope/reuse: changes remain in CASE translation/execution and tests; ColumnView, FunctionContext, Doris allocators and existing HashCRC32 are reused.
  3. Concurrency: fragment state is initialized before context cloning and read immutably; batch selections are local. No new threads or locks.
  4. Lifecycle: clones share fragment-state ownership and expression roots; literal children retain borrowed string storage. No ownership cycle or new static initialization dependency found.
  5. Configuration: default-off, forwarded session flag; short-circuit evaluation retains searched CASE.
  6. Compatibility: existing expression fields are reused. All BEs must be upgraded before enabling the experimental flag, as documented. No storage-format change.
  7. Protocol IDs: no Thrift/protobuf definitions changed.
  8. Parallel paths: searched/short-circuit fallback, linear/lookup matching, NULL/constant/selector inputs and nullable/complex results were checked. Shared constant-result broadcasting also affects the disabled-flag path and is included in the required CI coverage.
  9. Conditions/errors: comparison type/scale, common-input identity and literal eligibility are guarded; branch-index limits reserve zero correctly; duplicate matching THENs retain batch evaluation and child error propagation.
  10. Coverage: supplied tests cover thresholds, first matches, duplicates, NULLs, vector tails, wide indices, clones and fallback modes. The isolated result-assembly probe and oracle calculation passed; final Doris execution is pending CI.
  11. Expected results: the corrected oracle follows the same last-key remapping as CASE construction. No expected-result files were edited during this review; generation/execution provenance was not re-established locally.
  12. Observability: existing layout/debug output and Status/type checks are sufficient for this expression change. Repeated-limb correctness tests alone do not measure hash quality; collision counts/performance measurements remain useful follow-up evidence.
  13. Transactions/persistence: no transaction, visible-version, delete-bitmap, EditLog or failover change.
  14. Writes/atomicity: no persistent data-write path is changed; local result ownership and existing exception boundaries remain intact.
  15. FE/BE propagation: central expression serialization, layout flags, child ordering and clone paths carry the selected layout; unsupported inputs fall back.
  16. Performance/memory: matching removes equality-column materialization; literal lookup uses tracked containers and fixed-width selection buffers. Numeric result assembly can still scan rows per populated result column; further gather/SIMD comparisons require measurement.
  17. Additional review: no other substantiated blocking finding. Approval is for this exact reviewed head; ordinary branch protection and required build/test checks still apply.

…matching

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: Preserve eligible CASE expressions with a common input and literal keys behind a disabled-by-default experimental setting. Compare keys directly in BE, retain first-match and batch-level THEN evaluation semantics, and allow numeric matching loops to auto-vectorize. Short-circuit evaluation continues using searched CASE. Enable only after all BEs support the expression layout.

### Release note

Add the opt-in enable_experimental_simple_case setting, disabled by default.

### Check List (For Author)

- Test: BE/FE builds and targeted regression generation/rerun passed before the final loop simplification. Final actual-source optimized diagnostic compilation and SIMD reports verified; full rebuild and regression not rerun after the loop change. Unit tests added but not executed. Formatting, hygiene and targeted clang-tidy checked.
- Behavior changed: Yes, only when the experimental setting is enabled and short-circuit evaluation is disabled.
- Does this need documentation: No, experimental setting disabled by default.
### What problem does this PR solve?

Problem Summary: Matching each literal WHEN key scans the input batch repeatedly. Build a fragment-local lookup table for numeric and string keys above a branch threshold, preserve first-match results and batch evaluation of duplicate keys, and avoid expanding constant THEN columns. Extend branch indices beyond the former 256-branch limit.

An earlier INT/BIGINT/STRING implementation measured 256-branch mixed SQL speedups of 1.23x, 1.97x and 4.61x over linear matching on a shared machine. These measurements predate the final type, duplicate-key and allocator changes and are not validation of the final patch.

### Release note

Improve matching of eligible simple CASE expressions with many literal WHEN branches.

### Check List (For Author)

- Test: Earlier implementation passed BE UT (104), FE UT (11), related regressions and RELEASE builds; final changes have only clang-format, build hygiene and diff checks and have not been rebuilt or retested.
- Behavior changed: Yes; extend eligible simple CASE branch counts and internal numeric execution coverage.
- Does this need documentation: No
### What problem does this PR solve?

Problem Summary: The lookup regression records every input row for each combination, producing almost 40000 output lines. Aggregate row counts, non-NULL result counts and checksums of row IDs and results to retain coverage with 451 output lines.

### Release note

None

### Check List (For Author)

- Test: Generated output using run-regression-test.sh -d correctness_p0 -s test_simple_case_lookup -forceGenOut and reran normally; both passed against the existing 448cb83 service. All 75 setting pairs match. This does not validate the latest BE source.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Problem Summary: The default Decimal128 and Decimal256 hashes XOR independently hashed limbs. Distinct keys with repeated limbs therefore collide, making lookup initialization quadratic and missed probes linear in the number of WHEN branches. Use the existing HashCRC32 specializations for decimal keys and add repeated-limb Decimal128/256 cases with 256 and 1024 branches. Also correct the test oracle for the final duplicate key.

### Release note

Avoid pathological decimal-key hash collisions in experimental simple CASE lookup.

### Check List (For Author)

- Test: clang-format 16, build hygiene and git diff --check passed. Added unit tests have not been compiled or run.
- Behavior changed: No; matching results are unchanged.
- Does this need documentation: No
@Mryange

Mryange commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Related PR: apache#69296

Problem Summary: Including VExprContext in the simple CASE unit test raises the transitive reach of util/rle_encoding.h to 909 translation units, exceeding the build hygiene limit of 908 and failing CMake configuration. Extract ColumnIteratorOptions unchanged into a dedicated header and include it from VExprContext instead of column_reader.h. This reduces the RLE header reach to 272 translation units without increasing the budget.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, build-support/check-build-hygiene.sh and staged diff checks passed. Compilation and runtime tests not run; this change only relocates an unchanged options structure.
- Behavior changed: No
- Does this need documentation: No
@Mryange

Mryange commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Related PR: apache#69296

Problem Summary: Decoupling VExprContext from column_reader.h exposed callers that relied on its transitive includes. QueryContext uses a nested BlockFileCache type without its definition, and scanner, file reader and datetime code similarly require explicit type dependencies. Add the required direct includes and a cache forward declaration without changing execution logic or build hygiene budgets.

### Release note

None

### Check List (For Author)

- Test: ASAN BE build passed with ./build.sh --be -j48 (exit 0), including linking and packaging. clang-format 16, build hygiene and staged diff checks passed. Unit and regression tests not run.
- Behavior changed: No
- Does this need documentation: No
@Mryange
Mryange requested a review from Gabriel39 as a code owner October 27, 2026 04:09
@Mryange

Mryange commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28779 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7ae5d62e49f861e13c22b96b5235be45f4654a0, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17599	5160	5056	5056
q2	2126	337	294	294
q3	10206	1399	825	825
q4	4680	465	337	337
q5	7441	794	527	527
q6	178	186	152	152
q7	765	828	602	602
q8	9266	1338	1070	1070
q9	5465	4220	4184	4184
q10	6878	1288	1057	1057
q11	448	252	228	228
q12	635	411	278	278
q13	18095	3595	3040	3040
q14	281	275	263	263
q15	q16	726	731	686	686
q17	1196	759	573	573
q18	7002	6212	6149	6149
q19	1239	978	601	601
q20	387	343	223	223
q21	5604	2376	2445	2376
q22	402	330	258	258
Total cold run time: 100619 ms
Total hot run time: 28779 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5979	5903	6062	5903
q2	760	630	559	559
q3	5858	5862	5368	5368
q4	2005	2134	1601	1601
q5	5823	5915	5593	5593
q6	239	185	144	144
q7	2284	2019	1777	1777
q8	2973	2732	2707	2707
q9	7972	7519	7532	7519
q10	3663	3551	3125	3125
q11	592	424	375	375
q12	682	681	501	501
q13	3296	3619	2996	2996
q14	291	296	277	277
q15	q16	674	690	619	619
q17	7699	7080	6886	6886
q18	12959	12223	12910	12223
q19	884	764	773	764
q20	2151	2157	1913	1913
q21	5613	4601	4708	4601
q22	517	448	394	394
Total cold run time: 72914 ms
Total hot run time: 65845 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150964 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7ae5d62e49f861e13c22b96b5235be45f4654a0, data reload: false

query5	4324	593	445	445
query6	452	213	188	188
query7	4793	438	228	228
query8	321	170	145	145
query9	8764	3068	3094	3068
query10	485	305	250	250
query11	5942	3850	3578	3578
query12	148	94	85	85
query13	1249	431	365	365
query14	6109	3920	3626	3626
query14_1	3426	3376	3408	3376
query15	202	200	179	179
query16	945	436	392	392
query17	824	637	508	508
query18	2396	420	300	300
query19	183	155	131	131
query20	82	83	80	80
query21	209	125	103	103
query22	13033	13156	12912	12912
query23	13461	12805	12398	12398
query23_1	12550	12514	12467	12467
query24	7368	823	453	453
query24_1	482	473	482	473
query25	540	420	362	362
query26	1252	260	138	138
query27	2784	441	271	271
query28	4672	1932	1921	1921
query29	1571	591	450	450
query30	294	218	185	185
query31	1024	926	835	835
query32	145	93	95	93
query33	528	302	244	244
query34	996	845	492	492
query35	738	744	645	645
query36	829	814	723	723
query37	132	105	86	86
query38	1778	1733	1688	1688
query39	791	798	746	746
query39_1	737	726	742	726
query40	221	117	105	105
query41	75	71	68	68
query42	88	84	83	83
query43	340	359	315	315
query44	1278	685	693	685
query45	184	180	171	171
query46	811	966	562	562
query47	2892	2973	2713	2713
query48	323	322	223	223
query49	580	405	308	308
query50	684	290	220	220
query51	10549	10762	10593	10593
query52	80	85	78	78
query53	190	210	156	156
query54	247	204	186	186
query55	78	75	66	66
query56	232	237	226	226
query57	1658	1619	1571	1571
query58	291	278	267	267
query59	2180	2201	1925	1925
query60	310	259	237	237
query61	171	173	167	167
query62	398	338	281	281
query63	188	163	160	160
query64	2823	920	751	751
query65	3326	3264	3255	3255
query66	1774	408	300	300
query67	20307	20156	20125	20125
query68	3165	966	608	608
query69	383	287	231	231
query70	834	801	768	768
query71	285	248	214	214
query72	2823	2614	2266	2266
query73	506	520	299	299
query74	4697	4496	4333	4333
query75	2357	2298	1900	1900
query76	2322	1016	589	589
query77	342	362	276	276
query78	9882	9740	9169	9169
query79	979	844	509	509
query80	764	409	334	334
query81	588	314	271	271
query82	575	132	110	110
query83	310	193	177	177
query84	310	124	104	104
query85	863	427	400	400
query86	356	236	227	227
query87	2028	1948	1867	1867
query88	3638	2693	2649	2649
query89	327	292	243	243
query90	1782	164	163	163
query91	162	149	125	125
query92	99	87	90	87
query93	987	985	578	578
query94	507	318	270	270
query95	568	381	307	307
query96	670	486	233	233
query97	2450	2435	2318	2318
query98	161	156	147	147
query99	774	775	653	653
Total cold run time: 235159 ms
Total hot run time: 150964 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7ae5d62e49f861e13c22b96b5235be45f4654a0, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.04
query3	0.26	0.08	0.08
query4	1.61	0.09	0.10
query5	0.24	0.25	0.25
query6	1.17	0.68	0.68
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.35	0.36
query10	0.57	0.57	0.61
query11	0.19	0.13	0.14
query12	0.18	0.14	0.15
query13	0.49	0.49	0.49
query14	1.03	1.02	1.00
query15	0.62	0.60	0.60
query16	0.34	0.36	0.34
query17	1.12	1.11	1.12
query18	0.19	0.19	0.19
query19	2.09	2.06	2.04
query20	0.02	0.01	0.02
query21	15.48	0.19	0.14
query22	5.00	0.06	0.06
query23	16.27	0.33	0.12
query24	3.09	0.52	0.27
query25	0.11	0.08	0.05
query26	0.94	0.22	0.16
query27	0.05	0.04	0.03
query28	3.77	0.77	0.35
query29	12.53	4.21	3.26
query30	0.28	0.12	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.51
query33	3.10	3.20	3.27
query34	15.36	4.02	3.38
query35	3.34	3.33	3.32
query36	0.56	0.46	0.43
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 97.11 s
Total hot run time: 24.23 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 82.35% (28/34) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 38.00% (19/50) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Related PR: apache#69296

Problem Summary: Removing transitive storage dependencies from VExprContext exposed missing type definitions and a leaked RowRanges alias in unit tests. Add direct includes and use the existing native reader alias. Fix the simple CASE typed test's unused lambda capture and construct DecimalV2 fields with the required value type so all BE unit test sources compile.

### Release note

None

### Check List (For Author)

- Test: ASAN BEUT compilation and linking passed via run-be-ut.sh. CASE, index and Hudi tests passed (139 tests); ParquetV2NativeDecoderTest passed (119 tests), for 258 passing tests total. clang-format 16, build hygiene and diff checks passed.
- Behavior changed: No
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants