[fix](paimon) Harden JNI reader options and transport

[Gabriel39]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

The Paimon JNI read path had several correctness and operational guardrail gaps:

1. read.batch-size accepted zero, negative, or excessively large values. In particular, zero can make the vectorized reader return without advancing input.
2. scan.manifest.parallelism=0 can block on a zero-permit semaphore, while a value above the CPU-sized default causes Paimon to replace a JVM-static executor and leaves the new pool visible to later queries.
3. JNI projected fields used delimiter-based framing. A legal quoted column such as `region,code` was decoded as two fields, and nested STRUCT child names containing #, ,, or : also collided with the type grammar.
4. The legacy Paimon reader selected by enable_file_scanner_v2=false did not emit the safe schema payload used by FileScannerV2.
5. DEBUG logging printed the complete scanner parameter map, which may contain storage or JDBC credentials.
6. Catalog property validation temporarily published the candidate map. A concurrent first query could initialize from unjournaled values that were subsequently rejected and rolled back.
7. Catalog-level paimon.table-option.* accepted options which can select a different table context or affect writes after Doris has already bound the original schema.
8. Tightening that allowlist could make an existing catalog unloadable if its persisted image already contained a formerly accepted option.
9. Physical Paimon table options bypassed Doris catalog/relation validation and could still carry unsafe batch size, async threshold, or manifest parallelism into planning.
10. Every scanner statistics collection enumerated all JVM threads and allocated a ThreadInfo[], although the Paimon async-reader pool is JVM-global.
11. Safe reader tuning could not be supplied for an individual table@options(...) relation.

How does this PR solve it?

• Introduces one shared reader-option validator and allows these batch-read controls:
  • read.batch-size: 1..65536
  • file-reader-async-threshold: 1 MB..1 GB
  • file-index.read.enabled: Boolean
  • source.split.target-size: positive memory size
  • source.split.open-file-cost: non-negative memory size
  • scan.manifest.parallelism: 1..Runtime.availableProcessors()
  • scan.plan-sort-partition: Boolean
• Applies explicit precedence without mutating the physical Paimon table:
  relation @options > Doris catalog option > physical table option > Paimon default.
• Validates the final physical/catalog table view before planning or serialization. This prevents a table created with read.batch-size=0 or scan.manifest.parallelism=0 from bypassing Doris-side safeguards.
• Bounds relation-scoped and physical manifest parallelism to 1..Runtime.availableProcessors(), preserving the JVM-global executor-capacity invariant.
• Emits paired required_fields_base64 and columns_types_base64 schema payloads from both the legacy and V2 C++ readers. Every complete list item is independently encoded, and every nested STRUCT child name is encoded inside the type descriptor.
• Keeps the legacy required_fields and columns_types parameters as a compatibility fallback. The Java scanner accepts only a complete encoded pair, so it cannot combine two framing versions accidentally.
• Replaces raw parameter logging with a fixed summary containing only batch size and projected-field count.
• Adds detached catalog-property validation. Paimon validates a merged candidate CatalogProperty before the live catalog is changed; connectors without this capability retain the existing tentative-validation path and rollback behavior.
• Rejects new unsafe CREATE/ALTER settings, but load-time reconstruction ignores rather than applies unsupported or invalid legacy paimon.table-option.* entries. The raw persisted property is retained so image/edit-log compatibility is not destructive.
• Shares one async-reader thread-count sample across scanners for one second.

For example, the following nested schema is now transported without colliding with either the outer list framing or the STRUCT grammar:

CREATE TABLE quoted_fields (
  `nested#value` STRUCT<
    `hash#name`: STRING,
    `region,code`: STRING,
    `colon:name`: STRING
  >
);

Example query-level tuning:

SELECT id
FROM paimon_table@options(
    'read.batch-size' = '4096',
    'file-reader-async-threshold' = '32 MB');

Two aliases of the same table may use different values in the same statement because each relation receives its own Table.copy(...):

SELECT small.id
FROM paimon_table@options('read.batch-size' = '1') small
JOIN paimon_table@options('read.batch-size' = '8192') large
  ON small.id = large.id;

Thread statistics microbenchmark

Dataset and method:

• One JVM process per measurement.
• 16, 128, or 512 idle worker threads whose names match the Paimon async-reader prefix. Including JVM service threads, the observed live-thread counts were 22, 134, and 518.
• 200 warm-up calls followed by 2,000 measured statistics calls.
• Old path: every call executes getAllThreadIds() and getThreadInfo(ids, 0), then scans the returned ThreadInfo[].
• New path: the exact one-second shared-cache algorithm used by the scanner; the cache is primed before measurement.
• Elapsed wall time and bytes allocated by the calling thread were measured with ThreadMXBean. A checksum blackhole consumes every returned count.

Named workers	Live JVM threads	Old time / 2,000 calls	Cached time / 2,000 calls	Old allocated bytes	Cached allocated bytes
16	22	41.507 ms	0.429 ms	12,600,576	0
128	134	139.544 ms	0.417 ms	71,430,512	0
512	518	581.066 ms	0.398 ms	274,048,216	0

This microbenchmark isolates statistics collection rather than scan I/O. It shows that the old cost grows with JVM thread count, while cache hits avoid both the full thread walk and its per-call allocation. The tradeoff is that this profile gauge can be up to one second stale; it does not affect query execution or scheduling.

Tests

Passed locally:

• PaimonJniScannerTest: 15 tests, 0 failures, Maven build cache disabled.
• FE checkstyle: 0 violations.
• Doris C++ clang-format 16 check.
• Groovy compilation of the P0 regression suite.
• Paimon 1.3.1 compile/runtime probes covering bounds, physical manifest rejection, and legacy-option filtering.
• Thread-statistics microbenchmark described above.

Added coverage:

• BE unit tests for delimiter-safe JNI field-list framing and nested STRUCT type descriptors.
• Scanner unit tests for top-level and nested quoted identifiers, paired schema decoding, real DEBUG log capture/redaction, and shared cache TTL.
• FE unit tests for detached ALTER validation, a latch-controlled ALTER/initialization race, catalog/relation/physical option bounds, unsafe-option rejection, manifest parallelism, relation-local copies, and image/edit-log legacy-option reconstruction.
• P0 external-Paimon regression for both V1 and V2 scanner routes; quoted comma/hash/colon/space/Unicode identifiers; nested STRUCT projection; physical unsafe options; relation precedence; independent aliases; invalid boundaries; unsafe catalog options; and failed-ALTER rollback.

Not run locally:

• Full FE/BE compilation, FE/BE unit-test binaries, and the external-Paimon P0 environment. The checkout does not contain the required installed third-party toolchain (protoc included). CI buildall and regression checks are requested on this PR.

Release note

Paimon catalogs and relation @options now support seven bounded batch-read controls: read.batch-size, file-reader-async-threshold, file-index.read.enabled, source.split.target-size, source.split.open-file-cost, scan.manifest.parallelism, and scan.plan-sort-partition. Unsafe catalog, physical-table, and manifest-parallelism settings are rejected. Both Paimon scanner routes support quoted top-level and nested identifiers containing delimiter characters. Scanner DEBUG logs no longer include raw parameters, failed Paimon catalog validation is atomic, persisted legacy options remain loadable without being applied, and scanner statistics reuse a short-lived JVM-wide thread-count sample.

Check List (For Author)

• Test
  • Unit Test
  • Regression test case added
• Behavior changed:
  • Paimon dynamic reader settings now use an explicit safe allowlist and documented bounds.
• Does this need documentation?
  • Yes. Supported options, precedence, bounds, examples, compatibility behavior, and the statistics-cache tradeoff are documented in [docs] Document Paimon dynamic read options doris-website#4021 and above.

Review follow-up

The review fixes make empty identifiers and option precedence explicit:

• Every outer JNI schema token now carries a $ marker. Therefore an empty projection is encoded as the empty payload, while one empty field name is encoded as $; both the legacy and V2 C++ producers use this shared encoding.
• Effective reader options are validated only after the complete copy chain is formed. For example, physical read.batch-size=0 plus catalog 1024, or plus relation @options(...=4096), is accepted with the safe final value; the same physical table without an override is still rejected. The same rule applies to manifest parallelism.
• Added {} versus {\"\"} unit coverage, scanner constructor coverage, final-boundary/no-override coverage, safe catalog/relation override coverage, and P0 queries through both scanner routes. PaimonJniScannerTest now runs 15 tests with no failures. A direct Paimon 1.3.1 runtime probe also accepted the safe relation override and rejected the unsafe final value.

Review follow-up: hidden planning handles

This update closes the four remaining pre-ScanNode planning gaps:

• Read-only system-table wrappers: $partitions exposes an empty option map while its private data table performs manifest planning. Doris now applies and validates relation options on that hidden data table, while type inspection remains non-planning so a safe override is not rejected prematurely.
• Fallback branches: FallbackReadFileStoreTable is validated recursively because its main and fallback children plan independently.
• Snapshot partition binding: partition enumeration now uses the already merged effective table handle. A physical scan.manifest.parallelism=0 is rejected before enumeration, while @options('scan.manifest.parallelism'='1') is preserved instead of being lost through a raw catalog reload.
• Row counts and statistics: fetchRowCount() validates immediately before newScan().plan(); coverage exercises both ExternalRowCountCache and manual analysis entry points.

For example, given a partitioned table whose physical option is unsafe:

SELECT count(*)
FROM unsafe_table$partitions;
-- rejected before manifest enumeration

SELECT count(*)
FROM unsafe_table$partitions
@options('scan.manifest.parallelism'='1');
-- succeeds because the relation copy overrides the physical value

Added tests use real Paimon 1.3.1 PartitionsTable, FallbackReadFileStoreTable, and file-store table objects, plus a partitioned P0 fixture. FE checkstyle, Groovy compilation, and isolated Paimon fixture probes pass locally. Full FE unit tests and the external P0 environment remain delegated to CI because this checkout does not contain the installed third-party toolchain, including protoc.

Review follow-up: parser compatibility and wrapper identity

This update addresses four compatibility and planning-identity findings:

• Legacy STRUCT compatibility: the public unencoded parser again normalizes child keys to lowercase. For example, struct<Mixed:int> produces the historical key mixed; only the versioned Base64 Paimon grammar preserves Mixed.
• Privilege delegates: effective-table validation now walks every DelegatedFileStoreTable.wrapped() handle as well as the fallback child. Therefore a PrivilegedFileStoreTable(FallbackReadFileStoreTable(...)) cannot hide an unsafe fallback manifest executor.
• System-table identity: system wrappers are constructed with Paimon's SystemTableLoader directly from the same cached FileStoreTable that Doris validates. The catalog is no longer called a second time, so a refreshed physical handle cannot be planned while an older handle is validated.
• Reader-only projection reuse: relation options containing only metadata-neutral read controls (read.batch-size, file-reader-async-threshold, file-index.read.enabled, source.split.target-size, or source.split.open-file-cost) validate their effective table and then reuse the memoized latest snapshot projection. Snapshot-selecting and manifest/partition-order planning options retain the effective-handle projection path.

The partitioned counting test verifies that reader-only tuning performs zero direct projection loads, while the existing manifest-override test verifies that safety-sensitive options still load from the effective copy.

Validation for this update:

• Targeted legacy parser test: 1 passed, with recorded RED output [Mixed, UPPER] before the fix and GREEN output [mixed, upper] after it.
• PaimonJniScannerTest: 15 passed, confirming encoded nested names still preserve their exact spelling.
• Real Paimon 1.3.1 privilege-wrapped fallback probe: rejected the hidden unsafe fallback after the fix.
• FE and Java-extension checkstyle: 0 violations.

Full FE unit execution remains delegated to CI because the checkout does not contain the installed third-party toolchain, including protoc.

Review follow-up: late serialization and connector-scoped schema framing

This update closes the four findings from the latest review:

• System-table descriptor serialization is non-planning. Schema construction reads the cached raw wrapper only for its row type, so a safe relation override is not replaced by validation of the unsafe physical handle after binding.
• Memoized partition projection validates only manifest-planning settings. Reader batch and async settings are still validated on the final effective scan handle, but raw metadata loading no longer rejects values already replaced by a safe relation copy.
• V2 encoded schema framing is now an explicit Paimon reader capability. JDBC, Hudi, Trino, Iceberg system-table, and MaxCompute readers skip recursive encoded-type construction, Base64 encoding, and the two Paimon-only parameter-map entries.
• Every successful guardrail query now has a named result assertion. Coverage records quoted and nested values for both scanner routes, the empty identifier, catalog and relation overrides, alias isolation, failed-ALTER recovery, partitioned reader-option composition, manifest override, and system-table descriptor serialization.

For example, this composition now reuses the cached partition projection and validates the safe final scan handle:

SELECT *
FROM partitioned_table@options('read.batch-size'='4096');
-- physical read.batch-size=0 is ignored by metadata enumeration;
-- the effective relation copy is validated before data scanning

Validation for this update:

• PaimonJniScannerTest: 15 passed.
• FE checkstyle: 0 violations.
• Regression framework compilation and Doris clang-format 16 checks passed.
• FE/BE unit coverage and external-Paimon P0 result coverage were added but could not be executed locally because the installed third-party toolchain, including protoc, is absent; CI buildall and review were requested after the push.

Follow-up: complete safe batch-read options and BE UT compilation

This update expands the dynamic allowlist from two JNI-reader knobs to all seven safe batch-read controls consumed by the Doris Paimon 1.3.1 scan path:

Option	Accepted values	Doris use
read.batch-size	1..65536	JNI reader batch cardinality
file-reader-async-threshold	1 MB..1 GB	JNI asynchronous file-read threshold
file-index.read.enabled	Boolean	Paimon file-index pruning
source.split.target-size	Positive memory size	Target size for combining files into Doris scan splits
source.split.open-file-cost	Non-negative memory size	File-open cost used by split bin packing
scan.manifest.parallelism	1..Runtime.availableProcessors()	Manifest planning parallelism without mutating the JVM-global executor capacity
scan.plan-sort-partition	Boolean	Partition ordering during scan planning

The keys deliberately retain their official Paimon names. For example, read.batch-size must not become scan.read.batch-size: CoreOptions.READ_BATCH_SIZE reads the former key, while split controls belong to Paimon's source.* namespace and only manifest/partition planning controls use scan.*.

Catalog defaults and relation overrides accept the same list:

CREATE CATALOG paimon_hms PROPERTIES (
    "type" = "paimon",
    "paimon.catalog.type" = "hms",
    "paimon.table-option.source.split.target-size" = "64 MB",
    "paimon.table-option.file-index.read.enabled" = "true"
);

SELECT *
FROM paimon_table@options(
    'source.split.target-size' = '32 MB',
    'file-index.read.enabled' = 'false');

The existing precedence remains relation @options > Doris catalog > physical table > Paimon default. Relation-local split and reader settings can reuse the memoized partition projection because they do not select another snapshot; manifest parallelism and partition sorting keep an effective relation projection because they affect metadata planning.

The allowlist still excludes table-context selectors from Catalog defaults, scan.max-splits-per-task because it is consumed by Paimon's Flink source enumerator rather than Doris batch planning, and all streaming, storage-layout, write, and compaction settings. Snapshot/tag/startup selection remains available through the dedicated relation @options path.

The same commit fixes the BE UT compilation failure caused by two test-only JniColumn designated initializers omitting the newly added output_type member. Both aggregates now initialize it explicitly, preserving Clang's warnings-as-errors invariant.

Validation for this update:

• An isolated compile/runtime probe against Paimon 1.3.1 accepted all seven valid options and rejected invalid Boolean and size values.
• Regression framework compilation passed for 466 Groovy files.
• Doris clang-format 16 validation passed.
• FE unit tests, BE unit tests, and P0 external-Paimon coverage were expanded. Full local FE/BE execution was unavailable because the checkout lacks the installed third-party toolchain, including protoc.
• The latest TeamCity BE UT passed C++ compilation and entered GTest execution without the previous output_type designated-initializer diagnostics. The complete UT suite is still running.
• User-facing behavior, precedence, exclusions, and examples are documented in [docs] Document Paimon dynamic read options doris-website#4021.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Request changes: five blocking correctness and compatibility issues remain in the Paimon hardening paths.

Checkpoint conclusions:

• Goal and proof: the patch improves option validation, JNI identifier framing, raw-parameter log redaction, ALTER rollback, and thread-count sampling, but it does not accomplish the goal on all supported inputs and paths. The added tests do not cover the five blockers.
• Scope and clarity: the change is generally focused and reuses shared validators, although the generic CatalogMgr rollback change broadens the impact of connector-specific validation.
• Concurrency: the cached thread counter has sound volatile publication, synchronized refresh, retry, and wrap-safe TTL behavior. Failed ALTER is not atomic against concurrent lazy catalog initialization (MAIN-4).
• Lifecycle: Paimon Table.copy is relation-local and does not mutate cached base-table options; no C++ static-initialization-order or ownership issue was found. Failed ALTER can nevertheless leave initialized derived catalog state alive after raw-map rollback (MAIN-4).
• Dynamic options/configuration: catalog and relation reader overrides are observed without restart and their new bounds are clear, but equivalent values inherited from the physical Paimon table bypass validation (MAIN-3).
• Compatibility: retaining the legacy JNI field is rolling-upgrade-friendly only on the V2 producer. The still-supported V1 producer omits the safe payload (MAIN-1), nested type framing remains delimiter-unsafe (MAIN-2), and persisted pre-upgrade catalog options can prevent initialization (MAIN-5).
• Parallel and conditional paths: V2, V1/load routing, catalog, relation, and physical option sources were traced. The missed V1 and physical-table paths are blockers; the added validation conditions and error conversions are otherwise understandable.
• Tests/results: helper unit tests cover basic parsing, validation, redaction, and cache TTL. The P0 success blocks assert only absence of exceptions, leave Scanner V2 enabled, and omit nested names, unsafe physical options, a latch-driven ALTER race, and image/edit-log upgrade coverage. No generated expected-result file was added.
• Observability: raw scanner parameters are no longer logged and the cached JVM thread gauge is safe; no additional metric or log blocker was found.
• Persistence/transactions/data writes: edit-log ordering is correct in the single-threaded success/failure paths and no data-write transaction path changes here. Concurrent failed ALTER can expose unjournaled derived state (MAIN-4), while reload of existing persisted properties is incompatible (MAIN-5).
• Cross-module transport: no new FE-to-BE variable is introduced, but the BE-to-Java JNI schema parameter was updated in only one producer and its parallel type payload remains unsafe (MAIN-1 and MAIN-2).
• Performance: the JVM thread-count cache removes repeated thread walks without a material lock bottleneck. Unsafe physical manifest parallelism can still create a zero-permit wrapper or enlarge Paimon's JVM-static executor (MAIN-3).
• Additional user focus: no extra focus was supplied; the entire changed-file set was reviewed.

Validation was static only: the authoritative review prompt prohibits builds/tests, and this checkout lacks .worktree_initialized and thirdparty/installed. No build or test result is claimed.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 89.66% (52/58) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE UT Coverage Report

Increment line coverage 90.91% (10/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.04% (25557/43286)
Line Coverage	43.12% (256238/594184)
Region Coverage	38.81% (202996/523029)
Branch Coverage	40.14% (92549/230539)

[hello-stephen]

TPC-H: Total hot run time: 29850 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17811	4213	4136	4136
q2	2025	337	201	201
q3	10249	1400	816	816
q4	4683	476	361	361
q5	7545	858	574	574
q6	190	181	142	142
q7	755	811	618	618
q8	9331	1659	1683	1659
q9	5626	4430	4345	4345
q10	6717	1745	1482	1482
q11	512	360	321	321
q12	742	577	470	470
q13	18098	3430	2760	2760
q14	267	274	255	255
q15	q16	787	790	716	716
q17	989	1086	988	988
q18	7243	5751	5533	5533
q19	1289	1269	986	986
q20	802	689	598	598
q21	6196	2591	2583	2583
q22	420	357	306	306
Total cold run time: 102277 ms
Total hot run time: 29850 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4457	4354	4393	4354
q2	280	312	219	219
q3	4550	4948	4387	4387
q4	2063	2118	1380	1380
q5	4420	4280	4299	4280
q6	235	182	131	131
q7	1727	2331	1669	1669
q8	2574	2222	2184	2184
q9	8004	7973	7755	7755
q10	4646	4629	4213	4213
q11	619	460	391	391
q12	732	759	533	533
q13	3295	3643	2899	2899
q14	313	310	294	294
q15	q16	707	731	669	669
q17	1356	1359	1320	1320
q18	7976	7404	7181	7181
q19	1171	1110	1089	1089
q20	2195	2201	1933	1933
q21	5194	4523	4423	4423
q22	511	469	395	395
Total cold run time: 57025 ms
Total hot run time: 51699 ms

[hello-stephen]

TPC-DS: Total hot run time: 177722 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

query5	4330	644	481	481
query6	448	220	205	205
query7	4843	602	376	376
query8	332	198	171	171
query9	8783	4118	4105	4105
query10	466	360	304	304
query11	5873	2321	2120	2120
query12	159	101	98	98
query13	1280	629	458	458
query14	6255	5281	4981	4981
query14_1	4319	4271	4247	4247
query15	213	200	177	177
query16	1006	484	428	428
query17	929	681	567	567
query18	2428	474	362	362
query19	207	187	144	144
query20	111	109	106	106
query21	233	159	139	139
query22	13670	13600	13441	13441
query23	17501	16559	16155	16155
query23_1	16259	16150	16177	16150
query24	7512	1784	1290	1290
query24_1	1310	1305	1273	1273
query25	540	444	382	382
query26	1330	342	216	216
query27	2633	570	383	383
query28	4535	2032	2020	2020
query29	1083	672	464	464
query30	331	267	228	228
query31	1128	1091	993	993
query32	111	59	59	59
query33	535	304	259	259
query34	1174	1170	638	638
query35	763	788	687	687
query36	1041	1045	868	868
query37	146	106	92	92
query38	1861	1709	1653	1653
query39	876	871	844	844
query39_1	841	834	825	825
query40	253	164	142	142
query41	71	64	64	64
query42	92	91	89	89
query43	336	335	287	287
query44	1479	779	752	752
query45	194	180	171	171
query46	1053	1180	751	751
query47	2128	2079	1987	1987
query48	409	421	303	303
query49	567	418	315	315
query50	1043	427	353	353
query51	10723	10756	10631	10631
query52	85	86	75	75
query53	276	281	210	210
query54	287	261	240	240
query55	81	71	67	67
query56	326	286	283	283
query57	1303	1280	1208	1208
query58	282	256	252	252
query59	1596	1651	1448	1448
query60	303	276	247	247
query61	183	147	150	147
query62	538	497	433	433
query63	243	196	207	196
query64	2829	1029	847	847
query65	4741	4630	4655	4630
query66	1853	507	374	374
query67	29262	29211	29124	29124
query68	3046	1489	1032	1032
query69	399	316	268	268
query70	923	816	856	816
query71	374	351	323	323
query72	3080	2834	2519	2519
query73	842	765	462	462
query74	5080	4954	4710	4710
query75	2546	2526	2141	2141
query76	2317	1193	823	823
query77	366	387	293	293
query78	11867	11873	11284	11284
query79	1259	1200	786	786
query80	652	574	502	502
query81	494	338	303	303
query82	242	159	124	124
query83	330	340	301	301
query84	319	164	134	134
query85	973	606	527	527
query86	298	253	233	233
query87	1817	1822	1759	1759
query88	3740	2830	2784	2784
query89	414	383	333	333
query90	2137	208	201	201
query91	201	188	161	161
query92	65	62	53	53
query93	1517	1546	960	960
query94	536	366	301	301
query95	829	507	495	495
query96	1009	810	358	358
query97	2620	2602	2471	2471
query98	206	208	200	200
query99	1089	1106	987	987
Total cold run time: 261851 ms
Total hot run time: 177722 ms

[hello-stephen]

ClickBench: Total hot run time: 25.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b515efade9e8d84cb286c834e6708e0c65647e4, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.04
query3	0.28	0.14	0.14
query4	1.60	0.14	0.14
query5	0.24	0.22	0.22
query6	1.21	1.06	1.05
query7	0.04	0.00	0.01
query8	0.05	0.04	0.03
query9	0.38	0.30	0.31
query10	0.53	0.54	0.54
query11	0.19	0.13	0.14
query12	0.18	0.14	0.14
query13	0.48	0.47	0.48
query14	1.01	0.99	1.01
query15	0.62	0.60	0.63
query16	0.31	0.33	0.34
query17	1.12	1.13	1.09
query18	0.22	0.20	0.20
query19	2.09	1.96	1.91
query20	0.01	0.02	0.01
query21	15.45	0.20	0.13
query22	4.94	0.05	0.06
query23	16.13	0.30	0.11
query24	2.96	0.41	0.32
query25	0.10	0.05	0.05
query26	0.72	0.20	0.14
query27	0.03	0.04	0.04
query28	3.53	0.88	0.56
query29	12.47	4.12	3.33
query30	0.27	0.15	0.15
query31	2.77	0.57	0.31
query32	3.22	0.59	0.48
query33	3.18	3.14	3.23
query34	15.62	4.24	3.53
query35	3.51	3.55	3.56
query36	0.55	0.43	0.41
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.04	0.04
query40	0.19	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.65 s
Total hot run time: 25.02 s

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (11/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.64% (31937/42222)
Line Coverage	60.27% (355707/590220)
Region Coverage	56.91% (298647/524790)
Branch Coverage	58.27% (134367/230593)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 0.00% (0/58) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

run buildall

[Gabriel39]

/review

[hello-stephen]

TPC-H: Total hot run time: 29454 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18006	4149	4139	4139
q2	2322	327	197	197
q3	10296	1478	829	829
q4	4687	466	347	347
q5	7672	844	572	572
q6	194	171	140	140
q7	773	837	622	622
q8	9978	1442	1519	1442
q9	6693	4343	4308	4308
q10	7803	1712	1463	1463
q11	505	343	317	317
q12	745	608	473	473
q13	18114	3309	2713	2713
q14	267	258	255	255
q15	q16	789	781	709	709
q17	972	1047	1030	1030
q18	6787	5703	5473	5473
q19	1337	1244	1136	1136
q20	789	684	591	591
q21	5662	2570	2407	2407
q22	413	353	291	291
Total cold run time: 104804 ms
Total hot run time: 29454 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4413	4332	4314	4314
q2	283	319	206	206
q3	4530	4948	4376	4376
q4	2014	2153	1355	1355
q5	4426	4231	4448	4231
q6	231	266	218	218
q7	1788	1768	1467	1467
q8	2332	2006	2042	2006
q9	7687	7566	7685	7566
q10	4703	4615	4235	4235
q11	530	392	413	392
q12	766	752	541	541
q13	3222	3527	2991	2991
q14	294	319	280	280
q15	q16	721	733	637	637
q17	1352	1307	1401	1307
q18	7360	6899	6934	6899
q19	1119	1091	1112	1091
q20	2255	2247	1961	1961
q21	5549	4832	4895	4832
q22	539	475	444	444
Total cold run time: 56114 ms
Total hot run time: 51349 ms

[github-actions]

Two blocking correctness gaps remain:

1. The new Base64 list protocol still maps one empty top-level identifier to the same payload as zero projected fields, so the Java constructor rejects a reachable external Paimon schema on field/type cardinality.
2. Physical-plus-catalog options are validated before relation-scoped options are copied, so a safe relation override cannot take the documented precedence over an unsafe physical value.

Critical checkpoint conclusions:

• Goal and proof: the PR substantially adds bounded Paimon reader options, physical/catalog validation, relation-local copies, delimiter-safe transport, log redaction, detached ALTER validation, replay compatibility, and cheaper metrics. The two findings above prevent the full advertised identifier and precedence behavior. Existing tests do not cover either failing case.
• Scope and clarity: production changes are focused on Paimon/JNI behavior plus one generic ExternalCatalog validation hook and one shared JNI type parser extension. No unrelated source changes were found.
• Concurrency and locking: detached Paimon validation does not publish tentative CatalogProperty state; accepted updates reset synchronized ExternalCatalog objects and invalidate Paimon caches. The global thread counter uses synchronized refresh and volatile publication with intentional one-second staleness. No additional deadlock or stale-publication defect was found.
• Lifecycle and error handling: V1/V2 scanner construction and existing JNI cleanup remain aligned; mixed encoded schema halves fail loudly. Catalog rollback/replay/reset paths preserve prior or journaled state. No new resource leak or swallowed status was substantiated.
• Configuration and dynamic behavior: no Doris runtime config item is added. Catalog ALTER resets derived state, relation options use per-handle Table.copy, and unsafe values are range-checked; MAIN-2 is the remaining ordering error in the precedence chain.
• Compatibility and parallel paths: legacy schema parameters remain as fallback, V1 and V2 now emit the paired protocol, and persisted legacy catalog keys remain raw but inactive. Ordinary, branch, snapshot, system-table, cache-loader, planning, and JNI serialization paths were traced. MAIN-1 affects both scanner routes; MAIN-2 affects ordinary and system-table relation copies.
• Persistence, transactions, and writes: the catalog edit-log replay path was checked and retains loadability without applying unsafe legacy options. The PR does not alter Doris data transaction or storage-format semantics.
• Testing: review-time builds/tests were forbidden by the authoritative prompt and none were run in this review environment. CI was observed separately immediately before submission; its evolving state is not claimed as review-time test evidence. Changed tests cover the existing five review fixes but omit {} versus {""} framing and safe relation override of an unsafe physical value.
• Observability and performance: raw credential-bearing scanner parameter logging is removed, fixed summary fields remain, and the JVM thread-count sample removes repeated ThreadMXBean allocation. No additional observability or material performance regression was found.
• FE/BE propagation and nullability: both legacy and V2 BE producers publish the new paired Java parameters; no new FE-to-BE thrift variable or nullable-column manipulation is introduced.
• User focus: no additional user-provided focus was supplied; the full PR was reviewed.

The five existing inline threads were rechecked and not duplicated: legacy V1 producer parity, nested type framing, no-override physical option validation, detached ALTER validation, and persisted-catalog compatibility are addressed at the reviewed head.

[hello-stephen]

TPC-DS: Total hot run time: 177617 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

query5	4317	634	477	477
query6	515	240	203	203
query7	4935	591	351	351
query8	368	191	167	167
query9	8792	4027	4046	4027
query10	461	361	291	291
query11	6248	2361	2106	2106
query12	158	105	103	103
query13	1359	649	467	467
query14	6737	5280	4966	4966
query14_1	4342	4239	4229	4229
query15	217	212	189	189
query16	1040	475	441	441
query17	1130	706	581	581
query18	2684	468	350	350
query19	227	194	155	155
query20	118	110	109	109
query21	244	158	137	137
query22	13704	13610	13344	13344
query23	17375	16524	16716	16524
query23_1	16297	16406	16445	16406
query24	8069	1813	1343	1343
query24_1	1334	1366	1367	1366
query25	585	498	428	428
query26	1552	376	228	228
query27	2838	632	375	375
query28	4376	2012	1990	1990
query29	1073	579	467	467
query30	345	258	220	220
query31	1122	1084	969	969
query32	98	58	56	56
query33	530	312	250	250
query34	1223	1127	653	653
query35	831	769	655	655
query36	989	1024	885	885
query37	156	107	86	86
query38	1878	1726	1642	1642
query39	884	865	836	836
query39_1	839	864	846	846
query40	253	166	140	140
query41	65	61	60	60
query42	89	88	87	87
query43	338	340	278	278
query44	1418	759	742	742
query45	187	175	165	165
query46	1070	1250	751	751
query47	2080	2085	1930	1930
query48	380	406	281	281
query49	593	407	298	298
query50	1097	455	341	341
query51	10695	10703	10555	10555
query52	87	87	76	76
query53	254	268	201	201
query54	273	222	206	206
query55	73	74	66	66
query56	300	309	274	274
query57	1288	1305	1196	1196
query58	279	274	257	257
query59	1571	1676	1418	1418
query60	316	272	251	251
query61	154	149	150	149
query62	567	495	422	422
query63	236	197	195	195
query64	2825	1070	871	871
query65	4610	4478	4639	4478
query66	1827	492	366	366
query67	29429	29324	29105	29105
query68	3141	1583	981	981
query69	428	307	256	256
query70	945	816	805	805
query71	368	331	307	307
query72	3037	2690	2310	2310
query73	845	827	445	445
query74	5066	4877	4704	4704
query75	2532	2499	2148	2148
query76	2310	1158	794	794
query77	331	362	264	264
query78	11911	12016	11278	11278
query79	1364	1123	748	748
query80	1317	546	480	480
query81	541	333	287	287
query82	652	151	120	120
query83	382	327	290	290
query84	280	160	129	129
query85	965	622	547	547
query86	402	251	241	241
query87	1831	1829	1769	1769
query88	3700	2792	2746	2746
query89	427	384	322	322
query90	2007	202	194	194
query91	203	193	165	165
query92	61	59	55	55
query93	1648	1592	1012	1012
query94	704	351	313	313
query95	802	499	554	499
query96	1124	847	365	365
query97	2630	2615	2506	2506
query98	226	206	199	199
query99	1110	1114	966	966
Total cold run time: 266023 ms
Total hot run time: 177617 ms

[hello-stephen]

ClickBench: Total hot run time: 24.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b336eb4293f1857488e9d58dcdbf274ad908b05b, data reload: false

query1	0.01	0.00	0.00
query2	0.12	0.04	0.04
query3	0.25	0.13	0.13
query4	1.61	0.14	0.13
query5	0.24	0.21	0.21
query6	1.24	1.12	1.06
query7	0.03	0.00	0.00
query8	0.05	0.03	0.03
query9	0.37	0.31	0.30
query10	0.54	0.54	0.55
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.48	0.48	0.48
query14	1.01	1.03	1.01
query15	0.61	0.60	0.60
query16	0.31	0.32	0.31
query17	1.10	1.11	1.09
query18	0.22	0.21	0.20
query19	2.05	1.96	1.92
query20	0.02	0.01	0.01
query21	15.45	0.21	0.14
query22	4.82	0.06	0.05
query23	16.10	0.31	0.12
query24	2.95	0.45	0.32
query25	0.11	0.04	0.03
query26	0.74	0.21	0.16
query27	0.04	0.04	0.04
query28	3.48	0.91	0.55
query29	12.52	4.20	3.30
query30	0.27	0.16	0.17
query31	2.78	0.61	0.32
query32	3.22	0.60	0.47
query33	3.13	3.14	3.18
query34	15.59	4.24	3.51
query35	3.54	3.49	3.48
query36	0.56	0.43	0.42
query37	0.10	0.06	0.07
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.19	0.16	0.14
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.48 s
Total hot run time: 24.9 s

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Reviewed the full authoritative 22-file bundle; no additional user focus was supplied.

The four inline P1 findings are distinct remaining routes around the new effective-option guard: a read-only system-table wrapper hides its option-bearing data table, row-count planning runs outside PaimonScanNode, partition snapshot loading scans a raw physical handle before overrides, and a fallback reader exposes only the main branch's options.

Checkpoint conclusions:

• Goal and tests: direct data-table option precedence and both JNI scanner protocols are covered, but the changed tests do not exercise the four real wrapper/pre-planning/fallback routes above.
• Scope, parallel paths, and special conditions: V1/V2 JNI framing, data/system/incremental/options/no-options paths, empty and nested identifiers, partitioned metadata, row-count loading, and fallback branches were traced; only the four inline paths remain defective.
• Concurrency, lifecycle, persistence, and compatibility: detached catalog ALTER validation, single live publication/reset, replay/image compatibility, and legacy-key filtering are sound; no additional race or upgrade defect remains.
• Configuration and performance: valid catalog/relation precedence works on the foreground handle, but the four missed manifest paths can still block on a zero-permit executor or mutate Paimon's JVM-global manifest pool.
• Observability and FE/BE propagation: the credential-bearing raw parameter log is removed, the cached scanner metric is safely published, and the paired encoded schema plus serialized processed-table handoff are consistent.
• Data writes and transactions: no data-write/transaction behavior is changed by this PR, and no additional persistence issue was found.

Validation was static only. Per the authoritative bundle, no builds, tests, source changes, or commits were performed.

[Gabriel39]

run buildall

[Gabriel39]

/review

[hello-stephen]

TPC-H: Total hot run time: 29472 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ea521522915c65fcbcf1239de94d125d89c0ff9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17594	4220	4110	4110
q2	2035	326	197	197
q3	10281	1512	839	839
q4	4710	478	341	341
q5	7619	848	575	575
q6	211	173	139	139
q7	774	833	615	615
q8	10045	1615	1668	1615
q9	5775	4337	4340	4337
q10	6902	1765	1461	1461
q11	502	351	328	328
q12	743	577	458	458
q13	18107	3272	2678	2678
q14	267	263	241	241
q15	q16	773	784	727	727
q17	1046	1034	952	952
q18	6887	5918	5627	5627
q19	1322	1373	1125	1125
q20	804	683	542	542
q21	5639	2590	2249	2249
q22	434	359	316	316
Total cold run time: 102470 ms
Total hot run time: 29472 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4405	4303	4310	4303
q2	283	318	211	211
q3	4580	4956	4368	4368
q4	2057	2138	1371	1371
q5	4404	4244	4333	4244
q6	229	186	132	132
q7	1714	2053	1755	1755
q8	2451	2170	2112	2112
q9	7878	7750	7795	7750
q10	4693	4689	4245	4245
q11	572	421	384	384
q12	734	874	624	624
q13	3196	3510	3042	3042
q14	286	308	283	283
q15	q16	742	749	654	654
q17	1367	1332	1330	1330
q18	8035	7412	6957	6957
q19	1126	1114	1097	1097
q20	2242	2215	1939	1939
q21	5234	4603	4446	4446
q22	525	459	410	410
Total cold run time: 56753 ms
Total hot run time: 51657 ms

[github-actions]

Automated review summary

Requesting changes for four substantiated issues at 8ea521522915c65fcbcf1239de94d125d89c0ff9: three P1 correctness/compatibility gaps in legacy STRUCT parsing and effective Paimon wrapper validation, plus one P2 repeated partition-enumeration regression for reader-only OPTIONS.

Critical checkpoint conclusions:

1. Goal and proof — The guardrail, precedence, JNI framing, and credential-safe diagnostics goals are coherent and mostly implemented. The added FE/BE/P0 tests are broad, but they do not prove the four reported cases.
2. Scope and clarity — Most changes are focused and reuse common helpers. The shared ColumnType parser change unintentionally broadens the compatibility impact beyond Paimon, and the OPTIONS projection path does redundant work.
3. Concurrency — Detached catalog validation, synchronized publication/reset, double-checked system-table initialization, and volatile/synchronized counter sampling were traced. No lock-order or publication defect remains beyond the independently cached handle lifecycle reported inline.
4. Lifecycle — Scanner open/close and catalog reset lifecycles are balanced. The source-table cache and system-wrapper cache can nevertheless represent different generations, which is reported inline.
5. Configuration — Reader option ranges and relation > catalog > physical precedence are generally enforced, including CREATE/ALTER/replay paths. Privilege-wrapped fallback and divergent system-wrapper children bypass the intended complete effective-table validation.
6. Compatibility — The paired Base64 schema payload is emitted by both V1 and V2 Paimon readers and the encoded protocol is rolling-compatible. The unencoded public parser no longer preserves its prior lowercase STRUCT-key contract.
7. Parallel paths — Normal, snapshot, tag, branch, startup, incremental, row-count/statistics, partition, system-table, fallback, V1, and V2 paths were traced. The privilege-wrapped fallback topology is the remaining missed path.
8. Conditional checks — New fail-closed checks are generally documented and correctly placed; the direct instanceof FallbackReadFileStoreTable condition is insufficient for supported delegated wrappers.
9. Test coverage — Positive and negative unit/P0 coverage exists for ranges, precedence, lifecycle, framing, fallback, system tables, and row-count planning. Missing coverage corresponds to the four inline findings.
10. Test results — The changed expected behavior is internally consistent on inspection. No tests or builds were run because the authoritative review prompt prohibits them.
11. Observability — Raw scanner parameters are no longer logged; the derived debug summary and counters are appropriately bounded. No additional logging/metrics issue was found.
12. Persistence and transactions — Detached ALTER validation, EditLog publication/replay, restart behavior, and legacy property filtering were traced without a separate persistence defect. No data-write transaction path is changed.
13. FE/BE propagation — Both BE reader generations publish the new paired fields and the Java scanner consumes them consistently. The compatibility problem is confined to the shared legacy parser behavior described inline.
14. Performance — Reader-only OPTIONS unnecessarily bypass the memoized latest projection and re-enumerate partitions; no other material CPU, allocation, or memory-accounting regression was substantiated.
15. Other issues — External catalog input is trusted under the repository threat model; no separate security vulnerability was identified. All other candidates were either fixed in existing live threads or dismissed with concrete code evidence.

There was no additional user-provided review focus, so the whole PR was reviewed. All required second-round normal and risk reviewers converged on this exact four-comment set with NO_NEW_VALUABLE_FINDINGS.

[hello-stephen]

TPC-DS: Total hot run time: 177744 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ea521522915c65fcbcf1239de94d125d89c0ff9, data reload: false

query5	4327	635	478	478
query6	462	221	202	202
query7	4912	626	336	336
query8	344	189	179	179
query9	8774	4105	4092	4092
query10	482	358	319	319
query11	5945	2323	2133	2133
query12	158	103	106	103
query13	1264	617	450	450
query14	6247	5222	4873	4873
query14_1	4239	4229	4198	4198
query15	217	209	203	203
query16	1022	470	468	468
query17	1143	711	585	585
query18	2438	468	359	359
query19	209	189	150	150
query20	113	111	109	109
query21	247	161	137	137
query22	13543	13572	13411	13411
query23	17367	16480	16166	16166
query23_1	16241	16341	16290	16290
query24	7491	1763	1302	1302
query24_1	1310	1304	1270	1270
query25	577	461	398	398
query26	1328	361	222	222
query27	2593	607	382	382
query28	4432	2042	2074	2042
query29	1095	640	493	493
query30	347	275	230	230
query31	1122	1111	988	988
query32	111	64	62	62
query33	550	329	265	265
query34	1181	1098	665	665
query35	765	779	667	667
query36	1033	1019	912	912
query37	162	118	103	103
query38	1886	1700	1659	1659
query39	881	883	856	856
query39_1	842	848	870	848
query40	243	161	144	144
query41	66	63	65	63
query42	89	93	89	89
query43	321	320	281	281
query44	1398	772	773	772
query45	196	183	170	170
query46	1042	1200	734	734
query47	2122	2132	2050	2050
query48	398	416	303	303
query49	583	415	309	309
query50	1097	427	345	345
query51	10404	10793	10699	10699
query52	84	86	76	76
query53	273	279	196	196
query54	275	227	222	222
query55	73	72	65	65
query56	280	301	290	290
query57	1307	1289	1171	1171
query58	286	266	269	266
query59	1607	1657	1427	1427
query60	329	267	256	256
query61	149	154	153	153
query62	543	500	433	433
query63	248	198	202	198
query64	2856	1077	915	915
query65	4752	4665	4656	4656
query66	1837	495	380	380
query67	29484	29279	29139	29139
query68	3074	1537	1023	1023
query69	407	318	265	265
query70	889	798	808	798
query71	381	355	321	321
query72	3086	2680	2320	2320
query73	827	764	432	432
query74	5068	4910	4796	4796
query75	2525	2493	2148	2148
query76	2334	1195	778	778
query77	361	370	282	282
query78	12005	11921	11265	11265
query79	1421	1241	762	762
query80	1311	544	460	460
query81	521	331	284	284
query82	634	154	118	118
query83	360	327	293	293
query84	280	164	132	132
query85	964	625	516	516
query86	404	244	233	233
query87	1821	1818	1746	1746
query88	3734	2822	2815	2815
query89	448	400	326	326
query90	1901	196	193	193
query91	203	190	161	161
query92	62	60	59	59
query93	1622	1504	908	908
query94	715	362	288	288
query95	776	515	566	515
query96	1153	875	360	360
query97	2660	2602	2505	2505
query98	213	216	198	198
query99	1104	1128	975	975
Total cold run time: 263379 ms
Total hot run time: 177744 ms

[github-actions]

Static review found two blocking Paimon system-table schema consistency defects; both are annotated inline. They are inverse provenance failures: explicit historical OPTIONS lose their selected schema during runtime-safe source reconstruction, while statement-fence-derived reader OPTIONS incorrectly rewind system-table schema binding.

Critical checkpoints:

• Goal, scope, and focus: The PR coherently hardens Paimon reader-option validation, statement snapshot fencing, JNI schema transport, system-wrapper reconstruction, and plugin update validation. It does not fully accomplish the schema-consistency goal because of the two P1 paths. No additional user-provided focus was supplied; the full PR was reviewed.
• Concurrency and lifecycle: Scanner metrics/cached thread sampling, statement caches/reset, catalog ALTER locking, plugin TCCL restoration, wrapper/catalog-loader ownership, and scanner open/close paths were traced. No additional thread-safety, lock-order, lifecycle, or static-initialization defect was found.
• Configuration, compatibility, and parallel paths: Catalog/table/relation/BE option validation, dynamic catalog updates, API gates, V1/V2 encoded schema transport, legacy payloads, special conditions, and ordinary/fallback/delegated/system wrappers were checked. The two system-schema paths are the only remaining correctness gaps.
• Error handling, memory safety, and BE nullable handling: Status/exception boundaries and fail-loud guards were checked; the projection guard converts the first mismatch into a query failure rather than hiding it. No additional ignored-status, ownership/reservation, or ColumnNullable/const defect was found.
• Data correctness, persistence, and writes: The two inline findings are blocking data-read consistency issues. Catalog ALTER candidate validation plus journal/replay otherwise remain consistent; no transactional data-write path is changed.
• FE/BE propagation, performance, and observability: The encoded schema pair and Paimon side-channel options reach both JNI producer/consumer paths; branch-local manifest caps and cached JVM thread sampling are bounded; statistics, debug logs, and error context are adequate. No additional issue was found.
• Tests/results: Changed unit/regression coverage and expected results were inspected and are broad, but they miss the two cross-provenance cases called out inline. Static review only—no builds or tests were run under this review contract.

[github-actions]

[P1] Preserve historical schemas for explicit system-table OPTIONS. This branch uses copyWithoutTimeTravel for every marked OPTIONS pin, but a scan handle's systemTableSource is freshly resolved at the latest generation. After old is renamed to new, $ro/$audit_log@options('scan.snapshot-id'='S') binds old from S, then this rebuild exposes the latest row type to planScanInternal; its OPTIONS projection maps old to -1 and aborts before the backend rebuild can restore S. Planning-only OPTIONS do need latest-schema semantics, but explicit selectors need the selected schema. Please retain that provenance/source generation and cover split planning for a historical system view across a rename.

[github-actions]

[P1] Propagate this latest-schema sentinel to system-table binding. The normal getTableSchema arm honors schemaId=-1, but its system-table arm bypasses that check and resolveSystemTableAt always applies the marked OPTIONS map; the synthetic scan.snapshot-id=S therefore still rewinds the view's row type. After a schema-only add/rename following S, $ro@options('scan.manifest.parallelism'='1') binds S-era fields while the runtime-safe wrapper keeps the current schema. Please retain/honor the statement-fence provenance in the system path and cover both planning-only and explicit selectors.

[hello-stephen]

BE UT Coverage Report

Increment line coverage 71.01% (49/69) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.16% (25634/43327)
Line Coverage	43.24% (257278/594932)
Region Coverage	38.94% (203967/523810)
Branch Coverage	40.30% (93105/231057)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 95.65% (66/69) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.76% (32015/42258)
Line Coverage	60.44% (357166/590956)
Region Coverage	57.11% (300174/525577)
Branch Coverage	58.48% (135149/231095)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 23.85% (52/218) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28663 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24e69689de8f6ee32ec939867f9bdb8664ae1370, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17713	4004	3891	3891
q2	2023	308	193	193
q3	10276	1335	792	792
q4	4683	467	335	335
q5	7486	814	538	538
q6	189	164	134	134
q7	744	793	592	592
q8	9853	1530	1481	1481
q9	5883	3983	3991	3983
q10	6772	1611	1360	1360
q11	519	353	326	326
q12	742	570	458	458
q13	18122	3264	2756	2756
q14	268	264	249	249
q15	q16	737	729	657	657
q17	1035	914	960	914
q18	6742	5614	5500	5500
q19	1515	1314	1110	1110
q20	860	661	581	581
q21	5815	2660	2521	2521
q22	423	352	292	292
Total cold run time: 102400 ms
Total hot run time: 28663 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4345	4240	4183	4183
q2	289	321	208	208
q3	4525	4923	4347	4347
q4	2167	2284	1401	1401
q5	4264	4164	4117	4117
q6	222	173	127	127
q7	1705	1578	1542	1542
q8	2698	2167	2156	2156
q9	7247	7227	7292	7227
q10	4386	4279	3859	3859
q11	553	393	363	363
q12	704	732	505	505
q13	3192	3628	2881	2881
q14	289	303	275	275
q15	q16	679	721	659	659
q17	1320	1309	1365	1309
q18	7886	7200	7096	7096
q19	1074	1056	1031	1031
q20	2200	2198	1906	1906
q21	5244	4581	4393	4393
q22	505	472	406	406
Total cold run time: 55494 ms
Total hot run time: 49991 ms

[hello-stephen]

TPC-DS: Total hot run time: 169493 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24e69689de8f6ee32ec939867f9bdb8664ae1370, data reload: false

query5	4319	612	462	462
query6	471	220	207	207
query7	4850	577	319	319
query8	342	179	160	160
query9	8764	3996	3972	3972
query10	499	355	293	293
query11	5891	2185	2047	2047
query12	179	100	107	100
query13	1241	607	415	415
query14	6220	4665	4390	4390
query14_1	3805	3798	3799	3798
query15	210	212	168	168
query16	1000	466	483	466
query17	1092	665	531	531
query18	2564	459	333	333
query19	205	187	145	145
query20	103	102	99	99
query21	236	151	132	132
query22	12983	12897	12925	12897
query23	17279	16290	15959	15959
query23_1	16050	16117	16126	16117
query24	7440	1669	1224	1224
query24_1	1255	1241	1203	1203
query25	546	449	381	381
query26	1316	369	214	214
query27	2560	578	394	394
query28	4478	2030	2005	2005
query29	1070	631	504	504
query30	351	271	224	224
query31	1118	1096	970	970
query32	112	64	61	61
query33	537	328	255	255
query34	1175	1132	660	660
query35	730	740	649	649
query36	773	790	711	711
query37	165	113	90	90
query38	1825	1636	1619	1619
query39	825	831	792	792
query39_1	791	787	796	787
query40	253	171	151	151
query41	70	69	68	68
query42	95	93	92	92
query43	318	325	278	278
query44	1417	784	773	773
query45	199	176	196	176
query46	1053	1154	708	708
query47	1557	1508	1445	1445
query48	397	417	288	288
query49	581	399	287	287
query50	1125	416	343	343
query51	10790	10825	10793	10793
query52	83	85	79	79
query53	262	276	205	205
query54	291	228	219	219
query55	73	78	66	66
query56	300	310	275	275
query57	1010	1014	969	969
query58	276	247	243	243
query59	1534	1576	1390	1390
query60	306	272	272	272
query61	152	150	145	145
query62	404	326	270	270
query63	230	204	196	196
query64	2815	1007	813	813
query65	3921	3813	3869	3813
query66	1822	453	356	356
query67	28120	28196	27929	27929
query68	3314	1461	913	913
query69	415	308	273	273
query70	898	806	780	780
query71	367	351	343	343
query72	2955	2649	2333	2333
query73	837	733	423	423
query74	4684	4526	4277	4277
query75	2398	2335	2023	2023
query76	2315	1130	787	787
query77	335	393	268	268
query78	11045	11088	10516	10516
query79	1430	1129	763	763
query80	1302	545	463	463
query81	505	334	280	280
query82	627	147	115	115
query83	394	345	299	299
query84	307	165	138	138
query85	980	598	515	515
query86	390	225	226	225
query87	1790	1787	1752	1752
query88	3760	2826	2805	2805
query89	410	319	282	282
query90	1859	202	193	193
query91	203	193	164	164
query92	61	59	52	52
query93	1602	1575	1024	1024
query94	663	337	290	290
query95	773	579	467	467
query96	1030	812	358	358
query97	2471	2452	2329	2329
query98	206	208	192	192
query99	713	730	609	609
Total cold run time: 256100 ms
Total hot run time: 169493 ms

[hello-stephen]

ClickBench: Total hot run time: 23.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24e69689de8f6ee32ec939867f9bdb8664ae1370, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.25	0.13	0.14
query4	1.61	0.14	0.14
query5	0.25	0.22	0.22
query6	1.16	0.82	0.80
query7	0.03	0.00	0.00
query8	0.05	0.04	0.04
query9	0.37	0.32	0.32
query10	0.54	0.55	0.54
query11	0.19	0.13	0.13
query12	0.17	0.14	0.15
query13	0.47	0.47	0.48
query14	1.00	1.00	1.00
query15	0.61	0.59	0.60
query16	0.32	0.30	0.33
query17	1.11	1.12	1.10
query18	0.21	0.21	0.20
query19	2.07	1.92	1.89
query20	0.02	0.01	0.01
query21	15.43	0.19	0.15
query22	4.95	0.05	0.06
query23	16.13	0.30	0.12
query24	3.02	0.45	0.32
query25	0.11	0.04	0.05
query26	0.71	0.21	0.14
query27	0.03	0.03	0.03
query28	3.52	0.78	0.35
query29	12.51	4.01	3.18
query30	0.27	0.16	0.15
query31	2.77	0.58	0.32
query32	3.23	0.59	0.48
query33	3.27	3.18	3.18
query34	15.57	3.94	3.27
query35	3.26	3.24	3.22
query36	0.55	0.43	0.42
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 96.4 s
Total hot run time: 23.85 s

[Gabriel39]

run buildall

[Gabriel39]

/review

[hello-stephen]

TPC-H: Total hot run time: 28741 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a97eb74d5d6718c7a855b749bbfd9843e434d86, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17728	3925	3919	3919
q2	2068	320	203	203
q3	10257	1384	863	863
q4	4686	477	343	343
q5	7474	836	557	557
q6	183	169	138	138
q7	746	799	592	592
q8	9372	1545	1501	1501
q9	5288	4050	4002	4002
q10	6745	1611	1338	1338
q11	512	347	339	339
q12	724	576	450	450
q13	18068	3257	2761	2761
q14	262	270	245	245
q15	q16	746	732	659	659
q17	1004	974	1026	974
q18	7158	5712	5603	5603
q19	1379	1225	1015	1015
q20	789	653	598	598
q21	5865	2626	2348	2348
q22	449	357	293	293
Total cold run time: 101503 ms
Total hot run time: 28741 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4241	4162	4146	4146
q2	291	317	206	206
q3	4551	4959	4384	4384
q4	2188	2291	1407	1407
q5	4227	4145	4087	4087
q6	225	173	128	128
q7	1705	1565	1367	1367
q8	2800	2162	2080	2080
q9	7533	7441	7627	7441
q10	4318	4301	3942	3942
q11	557	418	358	358
q12	729	723	515	515
q13	3252	3488	2903	2903
q14	295	306	275	275
q15	q16	708	720	640	640
q17	1286	1272	1253	1253
q18	8098	7086	7351	7086
q19	1135	1118	1123	1118
q20	2226	2200	1963	1963
q21	5318	4615	4396	4396
q22	508	466	397	397
Total cold run time: 56191 ms
Total hot run time: 50092 ms

[hello-stephen]

TPC-DS: Total hot run time: 170185 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a97eb74d5d6718c7a855b749bbfd9843e434d86, data reload: false

query5	4307	614	478	478
query6	488	224	207	207
query7	4885	611	347	347
query8	347	191	172	172
query9	8790	4013	4026	4013
query10	455	344	282	282
query11	5834	2232	2024	2024
query12	154	99	100	99
query13	1259	589	429	429
query14	6098	4683	4400	4400
query14_1	3799	3840	3785	3785
query15	213	204	178	178
query16	1032	453	433	433
query17	943	698	563	563
query18	2430	458	353	353
query19	239	195	146	146
query20	103	104	100	100
query21	227	156	136	136
query22	13038	12916	12843	12843
query23	17332	16511	16097	16097
query23_1	16106	16149	16042	16042
query24	7631	1712	1231	1231
query24_1	1271	1212	1237	1212
query25	529	436	366	366
query26	1325	336	216	216
query27	2619	608	388	388
query28	4504	2078	2035	2035
query29	1068	605	476	476
query30	352	262	233	233
query31	1113	1074	957	957
query32	107	61	62	61
query33	521	305	234	234
query34	1184	1142	670	670
query35	728	746	619	619
query36	768	774	673	673
query37	152	104	91	91
query38	1832	1657	1603	1603
query39	848	825	818	818
query39_1	779	795	783	783
query40	260	169	137	137
query41	65	61	64	61
query42	94	89	89	89
query43	316	321	275	275
query44	1414	787	775	775
query45	201	176	167	167
query46	1067	1162	724	724
query47	1574	1489	1409	1409
query48	399	424	301	301
query49	590	395	308	308
query50	1100	425	361	361
query51	10698	10902	10762	10762
query52	86	90	74	74
query53	251	274	196	196
query54	278	251	216	216
query55	74	70	69	69
query56	312	294	281	281
query57	1025	1007	921	921
query58	287	273	252	252
query59	1499	1556	1371	1371
query60	315	278	260	260
query61	161	154	153	153
query62	393	325	260	260
query63	238	193	202	193
query64	2892	1203	989	989
query65	3910	3867	3784	3784
query66	1834	496	378	378
query67	28336	28283	28045	28045
query68	3161	1619	1023	1023
query69	436	314	281	281
query70	863	816	816	816
query71	372	345	324	324
query72	3410	2794	2287	2287
query73	843	837	439	439
query74	4636	4515	4279	4279
query75	2362	2336	2009	2009
query76	2300	1138	746	746
query77	336	368	272	272
query78	11212	11054	10609	10609
query79	1400	1134	752	752
query80	1287	536	467	467
query81	549	333	290	290
query82	583	155	115	115
query83	393	322	300	300
query84	329	166	134	134
query85	973	625	583	583
query86	406	224	229	224
query87	1820	1790	1686	1686
query88	3749	2834	2821	2821
query89	404	319	277	277
query90	1917	196	187	187
query91	202	199	161	161
query92	59	58	57	57
query93	1651	1597	935	935
query94	727	376	330	330
query95	778	566	566	566
query96	1099	784	346	346
query97	2504	2451	2334	2334
query98	204	201	198	198
query99	720	727	609	609
Total cold run time: 256902 ms
Total hot run time: 170185 ms

[hello-stephen]

ClickBench: Total hot run time: 23.84 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a97eb74d5d6718c7a855b749bbfd9843e434d86, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.14	0.14
query4	1.60	0.14	0.14
query5	0.23	0.21	0.22
query6	1.18	0.80	0.80
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.38	0.30	0.31
query10	0.56	0.58	0.55
query11	0.18	0.14	0.14
query12	0.18	0.15	0.13
query13	0.46	0.46	0.48
query14	1.02	0.99	0.98
query15	0.60	0.58	0.59
query16	0.34	0.32	0.31
query17	1.11	1.11	1.10
query18	0.22	0.19	0.20
query19	2.07	1.92	1.85
query20	0.02	0.02	0.01
query21	15.44	0.24	0.12
query22	4.81	0.06	0.05
query23	16.14	0.31	0.12
query24	2.95	0.46	0.35
query25	0.10	0.05	0.05
query26	0.74	0.22	0.16
query27	0.04	0.04	0.04
query28	3.55	0.77	0.37
query29	12.47	4.10	3.18
query30	0.26	0.16	0.18
query31	2.77	0.55	0.32
query32	3.22	0.59	0.49
query33	3.13	3.12	3.28
query34	15.77	3.94	3.28
query35	3.29	3.25	3.24
query36	0.56	0.45	0.44
query37	0.09	0.07	0.07
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 96.35 s
Total hot run time: 23.84 s

[github-actions]

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30750723456

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 61.93% (135/218) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 95.65% (66/69) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.84% (32049/42258)
Line Coverage	60.51% (357579/590956)
Region Coverage	57.13% (300268/525577)
Branch Coverage	58.55% (135303/231095)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 23.85% (52/218) 🎉
Increment coverage report
Complete coverage report

[github-actions]

PR approved by at least one committer and no changes requested.

[924060929]

Reviewed the current head 6a97eb7 with an FE-first focus and rechecked the previously reported paths against the latest code.

The end-to-end design is now coherent: relation options are applied after catalog options without mutating the cached physical table; validation runs on the final effective table; statement-level snapshot fences are shared across plain and option-bearing aliases; system-table, fallback, delegated, statistics, partition, prepared-execution, and deferred BE planning paths preserve the same option/snapshot/schema provenance; and the BE retains rolling-upgrade backstops for payloads from older FEs. The detached ALTER validation and connector classloader/API-major handling also close the catalog initialization race without making legacy images unloadable.

I did not find a new PR-introduced P1/P2 blocker on this head. The earlier hidden system-source runtime-cap and statistics-normalization findings are fixed and covered by targeted tests. Current FE/BE UT, external/P0 regressions, compile, style, and coverage checks are green.

The documented residual snapshot windows for a few Paimon system tables remain worth tracking separately: some Paimon SDK metadata paths ignore scan.snapshot-id, and the two-phase $files path cannot fully bind partition-marker enumeration to the older snapshot. These can cause transient system-table inconsistency during concurrent publication/partition deletion, but they are pre-existing SDK/path limitations, are explicitly outside the advertised OPTIONS capability matrix, and are not regressions introduced by this PR.

From the FE review perspective, this head is ready to merge. Given the size and cross-layer scope of the change, I recommend freezing this PR's scope and handling any further system-table consistency work in follow-up PRs.

1