[bobhan1]

What problem does this PR solve?

Issue Number: Part of #67611

Related PR: #67579 (validation and follow-ups), #67612 (infrastructure split)

Problem Summary:

Doris data pages are commonly much smaller than a 1 MiB File Cache block. On a cache miss, the existing query path expands a page read to the block boundary. This behaves like a fixed 1 MiB read-ahead for continuous scans, but the size and shape cannot follow the actual scan. Sparse point-query data-page reads can therefore download a full block for a small page, while sequential scans still expose one remote page dependency at a time through FileColumnIterator.

This PR adds an opt-in cloud query path that prepares each eligible top-level scalar scan column's compressed data pages once after pruning, advances its window on physical data-page reads, merges nearby pages into bounded file ranges, and serves the original exact page slices to the existing PageIO and decoder. Only a range that PageIO actually consumes becomes eligible for File Cache writeback. Complete cache blocks use the existing asynchronous cache writer; partial blocks enter a bounded background pipeline that fills their holes and then hands a complete block to the same writer.

The implementation covers top-level scalar columns in regular segment scans and short-circuit point queries, including scalar row-store reads. ARRAY/MAP/STRUCT and their offsets, null maps and children retain the original readers and legacy segment prefetchers; variant paths remain on the original path. TopN/global delayed-materialization second-stage read-ahead is outside the current PRs; that path stays identical to master. The feature is disabled by default and requires both enable_query_read_ahead=true and enable_async_file_cache_write=true. Planning, admission, asynchronous read, checksum, decode, and best-effort writeback failures preserve the original query result path.

Entry paths and component boundaries

flowchart TB
    SCAN["Regular SegmentIterator scan"]
    POINT["PointQueryExecutor<br/>row store or missing columns"]

    SCAN --> COLUMNS["After pruning: initialize scalar column windows once"]
    POINT --> LOOKUP["Collect exact row IDs and required scalar columns"]

    COLUMNS --> SEGMENT["SegmentReadAhead"]
    LOOKUP --> SEGMENT
    SCAN --> COMPLEX["Complex columns: original reader and prefetch"]
    POINT --> COMPLEX

    SEGMENT --> PLAN["FileRangePlanner"]
    PLAN --> SCHED["FileRangeReadScheduler"]
    SCHED --> POOL["Existing SegmentPrefetchThreadPool"]
    POOL --> READER["SegmentReadAheadFileReader"]
    READER --> PAGEIO["Existing PageIO and decoder"]
    READER --> WRITEBACK["Consumed-range writeback"]

Loading

Component	Responsibility
ColumnReadAhead / ColumnReadAheadPlan	Prepare eligible physical pages once and advance byte windows from page reads; the plan owns reset and timing aggregation.
SegmentReadAhead	Combine all column plans for a segment, probe Page Cache, plan ranges, submit one batch, and register exact page locations.
FileRangePlanner	Merge sorted page intervals and optionally complete high-coverage boundary cache blocks. It performs no I/O.
FileRangeReadScheduler	Admit ranges that fit query/BE budgets in one batch pass; provide cancellation and concurrent execution for already planned file ranges.
SegmentReadAheadFileReader	Serve an exact page slice from a ready range, immediately retire that page, and fall back to the original reader when needed.
RangeWritebackDispatcher / RangeCacheWriteback	Split a consumed range at 1 MiB cache-block boundaries and route complete blocks and partial fragments.
PartialBlockWritebackManager	Bound and deduplicate partial-block work, fill single ranges on block workers or multiple ranges on a shared remote-read pool, and submit complete owned buffers to the existing cache writer.

Read-ahead window

Columns are classified by dependency: eager columns have no earlier filtering dependency; lazy columns depend on an earlier filter or expression result. After segment pruning, initialize all currently resolvable eager and lazy windows together. Build each column's candidate-page indexes once from the pruned selection. Later data-page reads advance the window using the already resolved physical page index, before Page Cache lookup. Small scanner batches no longer rebuild rowid plans for every column.

Each window selects eligible pages until their compressed bytes reach the configured size; its final page can cross that size. The second selected page triggers the next window, or the only page for a one-page window. First access and a jump beyond planned coverage start at the current page. Reverse scans retain predictions while reading an ascending batch and retire them when the next lower batch begins.

Scalar point reads keep exact-row preparation, grouping sorted row IDs by distinct pages. Complex columns keep the original dependent-child reading path. Page completion updates ownership/accounting; the page-read position drives scan-window extension. Initial planning remains on the scanner thread; no separate planning pool is introduced.

Configuration	Default	Reason for the default
read_ahead_eager_window_bytes	4 MiB	An early trigger exposes the next window while the current one is read, bridging small scanner batches.
read_ahead_lazy_window_bytes	1 MiB	Expose more dependent-column I/O while retaining a smaller window than eager columns.

flowchart TD
    Init["After pruning: select candidate pages once<br/>initialize eager and lazy windows together"] --> Read
    Read["Column reader resolves a physical data page"] --> Advance["Retire passed predictions<br/>include the current page"]
    Advance --> Outside{"At or beyond planned coverage?"}
    Outside -- Yes --> Start["Start a byte window at this page<br/>mark its second selected page as the trigger"]
    Outside -- No --> Trigger
    Start --> Trigger{"Current page reaches the trigger?"}
    Trigger -- Yes --> Append["Append the next byte window<br/>move the trigger into it"]
    Trigger -- No --> Apply
    Append --> Apply["Submit newly planned pages<br/>reuse existing buffered pages"]
    Apply --> PageIO["Existing Page Cache / PageIO / decoder"]
    PageIO --> Read

Loading

The window is a planning unit, not a resident-memory limit; query and BE budgets bound retained range buffers. ReadAheadColumnInitTime covers initialization, ReadAheadPageAdvanceTime covers page-driven maintenance, and ReadAheadCurrentBatchPlanTime covers the retained exact-row path. Window discard and extension remain sub-timers under ReadAheadColumnPlanTime.

Range planning and concurrent I/O

The range merge uses a left-to-right algorithm inspired by Velox. Here, “merge” means combining nearby requested page intervals into one physical object-store GET. SegmentReadAhead supplies sorted, disjoint, non-empty page intervals; one linear pass adds the next interval only when every configured limit remains satisfied.

Configuration	Default	Meaning
read_ahead_max_gap_bytes	64 KiB	Largest unrequested gap that one foreground GET may cross.
read_ahead_max_range_bytes	2 MiB	Largest range produced by merging or foreground block completion. A single indivisible input page may exceed it and remains a standalone read.
read_ahead_max_read_amplification_ratio	1.0	Largest ratio of physical range bytes to requested page bytes.

After the base ranges are formed, FileRangePlanner examines their boundary File Cache blocks. read_ahead_block_fill_min_coverage=1.0 requires full requested-page coverage, so the default adds no foreground bytes to complete a partial block. Lowering this threshold retains the existing optional completion algorithm within the 2 MiB range limit.

flowchart LR
    PAGES["Sorted unique data-page intervals"] --> MERGE["Linear three-limit merge"]
    MERGE --> BASE["Base ranges"]
    BASE --> COVER{"Boundary block coverage<br/>at least 50 percent?"}
    COVER -->|"Yes and final range at most 2 MiB"| COMPLETE["Extend to block boundary"]
    COVER -->|No| KEEP["Keep base range"]
    COMPLETE --> BATCH["One atomic submission batch"]
    KEEP --> BATCH
    BATCH -->|Accepted| WORKERS["Concurrent range reads"]
    BATCH -->|Rejected| FALLBACK["Original synchronous path"]

Loading

FileRangeReadScheduler walks the planned ranges once under both query and BE budget locks, admitting each range that fits and continuing past skipped ranges so later smaller ranges can still proceed. Returned handles preserve the original range indices, with empty slots for budget skips; only admitted ranges receive buffers and tasks. It adds no range-count or concurrency quota. Accepted tasks run on the existing segment-prefetch thread pool, whose configured thread count controls physical read concurrency; the scheduler does not own or stop that shared pool. A byte reservation follows the query-owned buffer lifetime and is released with the last reference.

Scheduled reads suppress direct cache writes. A ready range is indexed back to every exact page it contains, so PageIO still receives the original offset and size and keeps the existing checksum and decoder behavior.

sequenceDiagram
    participant Scan as Scanner
    participant Column as ColumnReadAhead
    participant Segment as SegmentReadAhead
    participant Scheduler as FileRangeReadScheduler
    participant PageIO

    Scan->>Column: initialize candidates and first windows after pruning
    Column->>Segment: eager and lazy initial plans
    Segment->>Scheduler: page-cache probe, merge, complete blocks, submit ranges
    loop Physical data-page reads
        Scan->>Column: resolved page index
        Column->>Segment: new and released pages when the window changes
        Segment->>Scheduler: submit newly planned ranges
        Scan->>PageIO: exact page offset and size
        PageIO->>Segment: read buffered page
        Segment-->>PageIO: ready slice or original-reader fallback
    end

Loading

Point queries create one request-scoped range-read context and plan either the row-store column or all missing column-store values before decoding; existing row-cache and result-padding behavior remains unchanged. The point-query component owns the shared exact-row page helpers and their tests.

Consumed-range writeback

A ready range that read remote bytes becomes eligible for writeback when PageIO successfully reads its first registered page, and it is published at most once. Pure File Cache/inflight hits skip writeback. A prefetched range whose pages are all passed without use is released without a cache write.

flowchart TD
    CONSUMED["Consumed ready range"] --> ORIGIN{"Range read remote bytes?"}
    ORIGIN -- No --> SKIP["Skip duplicate writeback"]
    ORIGIN -- Yes --> SPLIT["Split at 1 MiB File Cache boundaries"]
    SPLIT --> FULL{"Fragment covers the complete valid block?"}
    FULL -->|Yes| ASYNC["Existing AsyncCacheWriteManager"]
    FULL -->|No| PARTIAL["PartialBlockWritebackManager queue"]
    PARTIAL --> DISPATCH["HoleFillDispatch: ordered deadline and capacity checks"]
    DISPATCH --> ELIGIBLE{"Aggregation deadline elapsed<br/>and cache writer has capacity?"}
    ELIGIBLE -- No --> WAIT["Dispatcher waits on condition variable<br/>task remains queued and mergeable"]
    WAIT --> ELIGIBLE
    ELIGIBLE -- Yes --> READY["Bounded worker handoff"]
    READY --> WORKER["Block worker: recheck cache and plan block-local holes"]
    WORKER --> COUNT{"Number of merged hole ranges?"}
    COUNT -- Zero --> OWNED["Complete owned block buffer"]
    COUNT -- One --> INLINE["Read directly on the block worker"]
    COUNT -- Multiple --> REMOTE["HoleFillRemoteReadPool: one task per range"]
    REMOTE --> JOIN["Block worker joins all accepted reads"]
    INLINE --> RESULT{"All reads succeeded?"}
    JOIN --> RESULT
    RESULT -- Yes --> OWNED
    RESULT -- No --> DROP["Drop best-effort writeback"]
    OWNED --> ASYNC
    ASYNC --> CACHE["Existing File Cache write path"]

Loading

Complete 1 MiB fragments, including a shorter physical EOF block, enter the existing asynchronous cache writer directly. Partial fragments are copied into a BE-level queue. Fragments for the same queued block merge under that task's fragment lock; an already active fill is deduplicated. A combined inflight-buffer and File Cache probe skips cached/downloading blocks before admission and once when a worker starts the task, outside the queue mutex. The task then executes its planned GETs and retains the final cache-writer checks. Its source pointer reads directly from the inner remote reader while retaining the cached wrapper as the shared owner, keeping the source open through delayed hole filling.

Queued partial blocks retain a mutable minimum aggregation interval, hole_fill_merge_delay_ms=2000 (milliseconds), to keep queued fragments mergeable for at least two seconds before dispatch. Timing starts at first admission; merging later fragments preserves the timestamp. The queue remains admission-time ordered, with stale replacements moving to the tail. One HoleFillDispatch thread checks due entries and stops at the first future deadline. It can bypass a due task whose writer lacks capacity, then waits until the earliest deadline or capacity retry. Runtime delay updates wake this dispatcher; tail arrivals that cannot advance its wait leave the timer alone. Waiting releases the mutex and preserves fragment merging. Complete blocks still enter the asynchronous cache writer directly.

The queue accounts one full cache-block buffer per queued or active entry and is bounded by hole_fill_max_pending_bytes_per_be=4 GiB. When full, the oldest queued task is evicted. If every retained task is already active, new cache-only work is rejected. The query read result is unaffected in either case.

Block workers, dynamically resized by hole_fill_workers_per_be=32, consume ready tasks. The total of dispatched and executing tasks is bounded by that worker count, so excess entries stay in the mergeable queue. Dispatch moves existing list nodes into the handoff queue. Shrinking lets already-dispatched work drain before admitting more. Writer-capacity accounting includes the handoff, and final write submission still uses spare-capacity-only admission. Active-task metrics include both dispatched and executing work.

Hole-fill algorithm

For one partial cache block, HoleFillPlanner unions the covered fragments, takes their complement inside that block, and applies the same three-limit merge algorithm to the missing intervals. Hole ranges never cross a 1 MiB block boundary. A single merged range is read directly on the block worker, avoiding task submission and wakeup overhead. Multiple ranges are submitted concurrently to the BE-wide HoleFillRemoteReadPool, controlled by mutable hole_fill_remote_read_threads_per_be=64. This pool limit applies only to multi-range reads; single-range reads also run on the block workers.

Each block worker creates and reuses one concurrent token for its lifetime. Before writeback or failure cleanup it joins all accepted GETs, including when a submission or read fails. Shutdown stops admission, joins the dispatcher and discards queued/undelivered work without its GETs, then joins executing block workers and releases their tokens before shutting down the remote-read pool. Block-worker and asynchronous cache-writer pools start with their configured worker counts; the remote-read pool retains at least one thread.

The hole-fill limits are tighter because crossing already covered bytes consumes background bandwidth without reducing the foreground query latency.

Configuration	Default	Meaning
hole_fill_max_gap_bytes	32 KiB	Largest already-covered gap crossed while combining adjacent holes.
hole_fill_max_range_bytes	1 MiB	Largest block-local hole-fill GET.
hole_fill_max_read_amplification_ratio	1.0	Largest ratio of physical GET bytes to missing bytes; the default forbids rereading covered gaps.

Example: all offsets below are KiB within one 1 MiB block.

• Covered fragments: [0, 256), [384, 416), [544, 1024)
• Complement: hole1 [256, 384), hole2 [416, 544)
• The holes are separated by 32 KiB of covered data. Merging would read 288 KiB for 256 KiB of missing data (1.125 amplification), exceeding the default limit of 1.0. Submit two parallel GETs, [256, 384) and [416, 544): 256 KiB total, with no covered bytes reread.

flowchart LR
    C1["covered 0 to 256 KiB"] --> H1["hole1 256 to 384 KiB"]
    H1 --> GAP["covered gap 384 to 416 KiB"]
    GAP --> H2["hole2 416 to 544 KiB"]
    H2 --> C2["covered 544 to 1024 KiB"]
    H1 --> GET1["GET1: 256 to 384 KiB"]
    H2 --> GET2["GET2: 416 to 544 KiB"]
    GET1 --> JOIN["Join both GETs"]
    GET2 --> JOIN
    JOIN --> BLOCK["complete block buffer"]
    BLOCK --> WRITER["existing async cache writer"]

Loading

Failure and compatibility boundaries

• The scheduler and hole-fill manager follow the Cloud BE lifetime. New query read-ahead paths require both enable_query_read_ahead=true and enable_async_file_cache_write=true; runtime updates affect newly initialized reads. Disabling asynchronous cache writes retains the original query reader and cache-filling path.
• The legacy segment File Cache prefetcher is skipped only after the new context is successfully created, avoiding duplicate prefetch work.
• Planning failure or a completely rejected submission releases the planned pages for the original reader. Partial budget admission registers only accepted ranges and completes skipped pages for synchronous fallback; only admitted, consumed buffers trigger writeback.
• A range read failure or cancellation retires its window entries and falls back when PageIO requests the page.
• If either enable switch closes during row-store point-read initialization, reuse the loaded segment with the original synchronous iterator.
• A checksum or decode failure on prefetched bytes evicts the suspect Page Cache entry and retries through the original reader.
• File Cache epoch changes, queue pressure, hole-read failure, and cache-writer rejection discard only best-effort writeback work.
• Variant-specific read-ahead forwarding remains outside this PR; existing Variant behavior is preserved.

Configuration defaults

Configuration	Default	Meaning
enable_query_read_ahead	false	Dynamically enable cloud query range read-ahead and consumed-range writeback for newly initialized reads; also requires enable_async_file_cache_write=true.
read_ahead_max_bytes_per_query	256 MiB	Maximum resident range-buffer bytes retained by one query.
read_ahead_max_bytes_per_be	1 GiB	Maximum resident range-buffer bytes retained across the BE.
read_ahead_eager_window_bytes	4 MiB	Compressed-page bytes selected for each eager-column window; its second selected page triggers the next window.
read_ahead_lazy_window_bytes	1 MiB	Compressed-page bytes selected for each dependency-delayed column window, using the same trigger rule.
read_ahead_max_gap_bytes	64 KiB	Foreground range-merge gap limit.
read_ahead_max_range_bytes	2 MiB	Foreground merged/block-completed range limit.
read_ahead_max_read_amplification_ratio	1.0	Foreground merge amplification limit.
read_ahead_block_fill_min_coverage	1.0	Minimum requested-page coverage for foreground cache-block completion.
hole_fill_max_gap_bytes	32 KiB	Background block-local hole-merge gap limit.
hole_fill_max_range_bytes	1 MiB	Background block-local GET limit.
hole_fill_max_read_amplification_ratio	1.0	Background hole-fill amplification limit; covered gaps remain unread by default.
hole_fill_max_pending_bytes_per_be	4 GiB	BE-wide memory bound for queued and active partial blocks.
hole_fill_merge_delay_ms	2000 ms	Mutable minimum time from successful admission before a queued partial block can activate; 0 disables it. Merges keep the original timestamp.
hole_fill_workers_per_be	32	Dynamically adjustable block workers for planning, single-range reads, joining multi-range reads, and writeback.
hole_fill_remote_read_threads_per_be	64	Dynamically adjustable shared pool size for parallel multi-range GETs; single-range reads execute on block workers outside this limit.

Read IO diagnostics

enable_read_io_trace=false dynamically enables actual S3 GET intervals/outcomes, range/task provenance, read-only cache/inflight coverage snapshots and typed timing events. Records are batched in memory and flushed by a dedicated writer to standalone JSONL files; they do not go to be.INFO and have no cumulative event cap. Normal and fast shutdown drain submitted records.

Configuration	Default	Meaning
enable_read_io_trace	false	Mutable diagnostic switch; leave disabled for performance measurements
read_io_trace_dir	empty	BE log directory's read_io_trace/ subdirectory, unless explicitly set
read_io_trace_flush_bytes	1 MiB	Memory-batch flush threshold
read_io_trace_flush_interval_ms	1000	Periodic flush interval

The offline analyzer reports interval union/extra transfers, within-source and cross-source overlap, submission stage totals/percentiles, queue-scan costs and GET latency. doris_read_io_trace_record_time_ns exposes diagnostic overhead. Capture and interpretation guide. The full implementation also records consumed-range writeback, queued/active fragments and dispatcher scan timing. Profile separates complete/partial foreground submission even with tracing disabled.

Commit structure

The branch is based on upstream master 33c460d7365 and contains 19 commits: 18 components and one unity-build fix. Its first seven carry the shared infrastructure tracked in #67612. Shared diagnostics enter once as component 6; commit 7 scopes filesystem error-code imports for no-PCH builds. Hole-fill and consumed-range follow-ups are folded into components 15, 17 and 18, so reviewers see their final form.

#	Component	Commit
1	Range coalescer	58e59000c8e
2	Cache-aware planner and shared metrics	a09525286ab
3	Asynchronous range scheduler	1b5b0a17942
4	Exact no-write cached reads	f32cb494d6a
5	Query runtime ownership	a0224415bca
6	Shared read IO trace and analyzer	823de9474d5
7	Scope filesystem error-code imports for unity builds	c7b4b8e0fa4
8	Page-driven column windows	bcb123b9a9e
9	Segment coordination / PageIO	6c694dfbc87
10	Scalar column-reader integration	aac13cb82de
11	Scalar scan initialization and legacy complex-column routing	8a4ab1d7ff0
12	Complete-block handoff and admission	8258390cc51
13	Cache-block range splitting	5780df79a85
14	Hole-fill planner	7a88d88a896
15	Partial-block writeback manager	831b4784e37
16	Hole-fill runtime wiring	175dda15981
17	Consumed-range writeback	0dd9f2f3db3
18	Query writeback wiring	faddd31829b
19	Point queries and exact-row helpers	bdc93f2a23c

Performance benchmarks

Time is FE Profile Total; S3 bytes include background I/O through drain.

Hardware

Item	FE	BE
Nodes	1	1
Instance	ecs.g9i.xlarge	ecs.g9i.8xlarge
CPU	Intel Xeon 6982P-C	Intel Xeon 6982P-C
vCPU / RAM	4 / 16 GiB	32 / 128 GiB
System disk	200 GiB	200 GiB
Data disks	1 × 300 GiB	4 × 800 GiB ESSD PL2

Test settings

Baseline is a cold read with asynchronous cache writes and page read-ahead disabled. Enabled is a cold read with asynchronous cache writes and page read-ahead/hole filling enabled. Local File Cache reads from warmed disk File Cache with both optimizations disabled and the OS page cache cleared.

Setting	Baseline	Enabled	Local File Cache
enable_async_file_cache_write	false	true	false
enable_query_read_ahead	false	true	false
File Cache	Empty	Empty	Warm
BE before each query	Restarted	Restarted	Restarted
OS page cache	Not explicitly dropped	Not explicitly dropped	Dropped

Key configuration	Enabled group
read_ahead_eager_window_bytes / read_ahead_lazy_window_bytes	4 MiB / 1 MiB
read_ahead_max_gap_bytes / read_ahead_max_range_bytes	64 KiB / 2 MiB
read_ahead_max_read_amplification_ratio / read_ahead_block_fill_min_coverage	1 / 1
read_ahead_max_bytes_per_query / read_ahead_max_bytes_per_be	256 MiB / 1 GiB
hole_fill_merge_delay_ms	2000 ms
hole_fill_max_pending_bytes_per_be	4 GiB
hole_fill_max_gap_bytes / hole_fill_max_range_bytes	32 KiB / 1 MiB
hole_fill_max_read_amplification_ratio	1
hole_fill_workers_per_be / hole_fill_remote_read_threads_per_be	32 / 64
async_file_cache_write_workers_per_disk / async_file_cache_write_max_pending_bytes	16 / -1 (automatic)
segment_prefetch_thread_pool_thread_num_min / segment_prefetch_thread_pool_thread_num_max	32 / 2000
file_cache_each_block_size	1 MiB
disable_storage_page_cache / storage_page_cache_limit	false / 20%
enable_query_segment_file_cache_prefetch / enable_compaction_segment_file_cache_prefetch	false / false
enable_read_io_trace	false
enable_sql_cache / enable_query_cache	false / false
parallel_pipeline_task_num	0 (automatic)

Summary

Benchmark	Queries	Baseline (s)	Enabled (s)	Local File Cache (s)	Baseline / Enabled
SSB Flat SF1000	13	767.878	498.556	159.685	1.54×
TPCH SF1000	22	2405.090	1224.448	542.209	1.96×
SSB SF1000	13	1645.817	597.873	298.043	2.75×

Benchmark	Baseline S3 (GiB)	Enabled S3 (GiB)	Change	Local File Cache S3 (GiB)
SSB Flat SF1000	255.118	264.763	+3.78%	0.000
TPCH SF1000	859.982	908.671	+5.66%	0.000
SSB SF1000	457.434	467.413	+2.18%	0.000

SSB Flat SF1000: per-query results

Query	Baseline (s)	Enabled (s)	Local File Cache (s)	Baseline S3 (GiB)	Enabled S3 (GiB)	Local File Cache S3 (GiB)
q1.1	16.024	12.499	2.706	4.009	4.056	0.000
q1.2	2.667	1.973	0.316	0.598	0.598	0.000
q1.3	7.367	5.359	1.687	1.560	2.028	0.000
q2.1	101.000	69.000	21.226	33.437	34.762	0.000
q2.2	88.000	66.000	18.662	30.061	30.700	0.000
q2.3	89.000	64.000	18.571	30.060	30.698	0.000
q3.1	81.000	58.410	17.170	27.608	29.360	0.000
q3.2	99.000	60.000	21.089	33.836	35.629	0.000
q3.3	79.000	49.984	16.107	27.837	28.730	0.000
q3.4	3.398	3.008	0.350	0.868	0.894	0.000
q4.1	126.000	70.000	26.058	41.061	42.393	0.000
q4.2	35.946	19.131	7.233	11.255	11.638	0.000
q4.3	39.476	19.192	8.510	12.929	13.277	0.000
Total	767.878	498.556	159.685	255.118	264.763	0.000

TPCH SF1000: per-query results

Query	Baseline (s)	Enabled (s)	Local File Cache (s)	Baseline S3 (GiB)	Enabled S3 (GiB)	Local File Cache S3 (GiB)
q1	123.000	55.869	32.193	36.204	37.280	0.000
q2	24.222	21.871	3.841	5.608	6.921	0.000
q3	139.000	82.000	24.373	45.455	47.923	0.000
q4	36.428	28.795	26.139	46.518	51.488	0.000
q5	221.000	102.000	47.868	76.312	78.798	0.000
q6	36.233	25.955	3.803	12.173	12.173	0.000
q7	118.000	69.000	19.579	41.299	42.104	0.000
q8	372.000	112.000	61.000	96.768	100.420	0.000
q9	349.000	133.000	75.000	104.408	109.942	0.000
q10	74.000	40.539	28.894	49.902	54.172	0.000
q11	18.481	11.135	3.591	5.293	11.127	0.000
q12	24.542	19.707	7.801	25.759	27.577	0.000
q13	144.000	57.676	23.791	34.590	37.258	0.000
q14	25.459	20.422	2.096	8.149	8.514	0.000
q15	26.637	18.529	2.872	9.000	9.136	0.000
q16	12.482	11.942	4.602	3.684	4.618	0.000
q17	148.000	100.000	32.692	50.098	50.552	0.000
q18	52.004	50.487	38.521	52.222	56.472	0.000
q19	170.000	84.000	34.844	58.621	61.535	0.000
q20	49.088	26.180	10.383	21.127	21.652	0.000
q21	198.000	120.000	52.502	62.617	64.747	0.000
q22	43.514	33.341	5.824	14.177	14.263	0.000
Total	2405.090	1224.448	542.209	859.982	908.671	0.000

SSB SF1000: per-query results

Query	Baseline (s)	Enabled (s)	Local File Cache (s)	Baseline S3 (GiB)	Enabled S3 (GiB)	Local File Cache S3 (GiB)
q1.1	38.378	12.701	3.472	5.216	5.272	0.000
q1.2	17.907	13.255	3.509	5.079	5.250	0.000
q1.3	32.784	11.881	3.523	5.079	5.250	0.000
q2.1	242.000	69.000	36.688	56.297	56.979	0.000
q2.2	272.000	69.000	36.757	56.297	56.909	0.000
q2.3	185.000	71.000	36.774	56.293	57.149	0.000
q3.1	158.000	64.000	31.572	48.839	49.388	0.000
q3.2	164.000	62.000	31.556	48.851	49.387	0.000
q3.3	152.000	65.000	30.527	48.768	49.931	0.000
q3.4	29.399	17.252	3.958	7.772	8.404	0.000
q4.1	236.000	99.000	53.401	80.008	83.789	0.000
q4.2	59.550	22.417	13.188	19.480	19.909	0.000
q4.3	58.799	21.367	13.118	19.456	19.795	0.000
Total	1645.817	597.873	298.043	457.434	467.413	0.000

Release note

Add an opt-in range read-ahead and partial File Cache block hole-fill pipeline for cloud query data pages. The feature is disabled by default.

Check List (For Author)

• Test:
  • Performance benchmark results are summarized above.
  • 141 tests / 8 suites passed with run-be-ut.sh -j100 --run (ASAN/PCH); the direct Release/PCH BE build passed with build.sh --be -j100.
  • Regression test
  • Unit Test
  • Manual test
  • No need to test or manual test.
• Behavior changed:
  • No.
  • Yes. When explicitly enabled, eligible cloud query data pages use bounded asynchronous range read-ahead and consumed-range File Cache writeback. The feature remains disabled by default; opt-in reads use the defaults listed above.
• Does this need documentation?
  • No.
  • Yes. Public configuration documentation remains pending for the split delivery; [Tracking] PageIO read-ahead, I/O coalescing, and cache block hole filling #67611 tracks the components and current configuration.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[github-actions]

Static review of all 73 changed paths at commit 990f3eb: changes requested. One new P1 identifies a BE abort when consumed-range writeback encounters supported irregular S3 upload cache blocks. Existing blocking comment 4278847468 still applies to the row-store point-query helper at base_tablet.cpp:175: a concurrent mutable-switch disable makes create_for_query return OK/null before its unconditional assertion. Its regular-scan instance is fixed. The pruned-map-child P1 (4278847462) and delayed-reader-owner P1 (4278847472) are fixed at this head and are excluded from existing_blocking_comment_ids.

Existing nonblocking reports remain applicable and are not reposted: queued-fragment metadata growth (4278847476), lost workload-group IO context (4278847479), dictionary-page corruption recovery (4324631642), bypassed query cache-write admission (4324631645), and packed-file trace lifecycle linkage (4324631648). The old read-ahead-only cache-population issue (4278847474) is fixed by requiring async cache writing in enabled().

The reviewed flow is scalar scan/point row IDs -> column byte windows -> segment page-cache probe and range planner -> query/BE byte admission -> shared prefetch workers -> exact PageIO slices/checksum/decoder -> consumed-range dispatch -> complete writer or bounded partial-block workers -> epoch-fenced cache persistence.

• Goal and proof: the opt-in scalar cloud read-ahead and consumed-range writeback flow is implemented, with the blocking and existing recovery/admission exceptions above. Added unit tests exercise range/window planning, scheduling and decoding, but do not prove the reported irregular-cache and row-store toggle cases.
• Scope, clarity and reuse: responsibilities are separated among column windows, segment coordination, range planning, scheduling and writeback; complete blocks reuse the existing cache writer. The change is large but stays within this feature, its profiles and diagnostics.
• Concurrency and locks: foreground windows are scanner-local; shared budgets, read states, page registrations and queues use their respective locks. Waits and consumer callbacks release the page-registry mutex; activation plus the fragment mutex stabilizes hole coverage; remote tokens join accepted reads. No additional lock inversion was substantiated. Workload attachment remains the existing P2.
• Lifecycle and static initialization: shared handles retain range buffers, budgets and source readers; aliasing ownership now retains the cached wrapper through delayed fills. Foreground-only callbacks keep borrowed column/statistics access within foreground ownership. Shutdown drains managers before dependent pools/cache destruction. No new cross-translation-unit initialization dependency was identified.
• Configuration: the feature requires cloud mode and both enable switches. Window/range policies are captured per coordinator; worker counts and merge delay have runtime callbacks. Scan initialization handles OK/null after a mutable disable; the point row-store helper still does not. Supported S3 upload-buffer sizing exposes the new P1.
• Compatibility: no wire or persisted segment format is changed. Existing cache intervals must remain consumable across configuration changes and restarts; M1 violates that compatibility. No compiled ELF/glibc compatibility audit was performed in this static review.
• Thrift/protobuf: no definitions or numeric IDs change.
• Parallel paths: nested, extracted and variant columns retain original readers/prefetchers; compaction/local and delayed-materialization second-stage paths retain their established routing. Prepared scalar point iterators preserve rowset version, commit TSO and absent-column defaults.
• Conditional checks: role vectors are disjoint by earliest materialization assignment. Enabled gates and fallback checks were traced to their callers; the remaining row-store assertion and strict cache-shape probe have concrete failure scenarios.
• Tests and negative coverage: inspected all changed test bodies for EOF, sparse/reverse scans, budget rejection, short/failed reads, cancellation, epoch changes, merge/activation, resizing and shutdown. Irregular upload cache layouts, row-store switch races and dictionary cache corruption are missing negative cases. No builds or tests were run under the review-only contract.
• Test results: no regression output files change; no runtime passing result is claimed.
• Observability: shared counters cover planning, source IO, fallback, consumption and writeback; trace diagnostics include GET/range/task IDs and asynchronous tails. Packed-file lifecycle joins remain the known P3. Trace buffering is explicitly documented as uncapped diagnostic behavior.
• Transactions and persistence: transaction visibility, rowset versions, delete bitmaps, EditLog and failover protocols are unchanged; read paths retain selected rowset context.
• Data writes and atomicity: writes populate disposable File Cache through existing writer/finalize semantics and invalidation epochs. The new pre-write probe can abort before best-effort persistence; cache-write budget and metadata bounds retain their existing reported gaps.
• FE/BE variable passing: no new transmitted variables or FE send paths.
• Performance and memory: scan candidate metadata is prepared once and advanced by physical page; range buffers follow explicit query/BE byte budgets, partial buffers follow block-count limits. Repeated fragment metadata remains the known independent bound violation; no additional concrete performance defect was substantiated.
• User focus and remaining limits: no additional user focus was supplied. All changed paths and initial risks have evidence-based dispositions. Findings are based on static control-flow inspection; runtime validation remains outside this invocation.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #67292 (comment)

[github-actions]

[P1] Handle irregular cached upload blocks before probing

Supported S3 upload settings can leave cache blocks that are not aligned to the global block grid: s3_write_buffer_size=5.5 MiB with 1 MiB cache blocks makes S3FileWriter populate intervals such as [5,5.5) and [5.5,6.5) MiB. After the former is evicted, consuming a read-ahead range such as [5.25,5.75) fetches remote bytes and submits a partial fragment for [5,6). This call then reaches BlockFileCache::probe(), whose DORIS_CHECK requires the resident block to start at 5 MiB; finding the retained 5.5 MiB block aborts the BE. Native queries with peer reads enabled previously used the synchronous, interval-aware get_or_set() path even with async writes enabled, so enabling read-ahead newly exposes this crash. Use an interval-aware coverage check here and cover partially evicted, non-multiple S3 upload layouts.

[bobhan1]

run buildall

[bobhan1]

/review

[github-actions]

Static review of all 74 changed files at bdc93f2. Request changes: three new P2 findings are inline (two memory issues and an executor-failure abort), and existing P1 4344901080 remains blocking. The latter was independently traced from non-block-aligned S3 upload cache cells through partial writeback to the grid-only probe assertion; it is not reposted.

Existing threads were revalidated. The delayed-reader ownership, mutable-switch initialization, pruned-map-child abort, and read-ahead-only/async-writer-disabled issues are repaired. The previously reported queued-fragment metadata growth (4278847476), workload-group IO bypass (4278847479), dictionary corruption recovery (4324631642), query cache-write budget bypass (4324631645), and packed-file diagnostic lifecycle mismatch (4324631648) still apply and are not duplicated.

There was no additional user-provided focus. Applicable critical checkpoints:

• Goal and evidence: scalar scans and point reads connect column windows to range planning, asynchronous reads, exact PageIO slices, and consumed-range writeback. Unit sources exercise the main mechanisms; the new memory findings and existing blocker prevent a clean verdict.
• Scope, clarity, reuse: column planning, segment coordination, scheduling, and block writeback have separate responsibilities; original decoders and the existing cache writer are reused. Full-index copies per iterator add the memory cost identified inline.
• Concurrency: scanner submissions, shared read workers, hole dispatcher/block/remote workers, and cache writers were traced. Budget locks, read-state mutexes, fragment activation, and lifecycle flags provide the reviewed coordination; no additional concrete lock inversion was found. Shared-executor callback retention after submission failure is a new inline finding; workload-group context remains the existing finding.
• Lock weight and order: partial-block source IO and full cache probes run outside its queue mutex; fragment copies use a separate mutex. Lifecycle-to-queue ordering and race-safe final writer admission were checked.
• Lifecycle and static initialization: aliasing ownership now retains the cached wrapper through delayed hole reads; cancellation uses weak read registrations, and shared statistics survive scanner teardown. Shutdown joins work before destroying its cache/executor owners. No additional static-initialization defect was established.
• Configuration: the feature is disabled by default and requires both enable switches in cloud mode. Validators and mutable worker/read-pool/delay callbacks were inspected; all factory OK/null callers now fall back when switches close.
• Compatibility: no wire, storage-format, or persisted-schema change was introduced. Binary ABI/ELF compatibility was not executed or asserted.
• Thrift/Protobuf field IDs: not applicable; no gensrc definitions changed.
• Parallel paths: scalar regular/point/row-store reads, forward/reverse batches, sparse selection, constants, schema defaults, version/TSO values, and Page Cache were traced. Complex/variant/extracted paths retain the original readers; TopN/global delayed-materialization second-stage read-ahead remains outside this feature.
• Conditional checks and errors: optional planning/admission/read/data-page decode failures use the original path. The irregular-block assertion and dictionary recovery exception remain existing threads; the new failed-submission state transition can also abort the BE as explained inline.
• Test coverage: added C++ tests cover planners, admission, cancellation, epochs, merges, worker races, source retention, fallback, roles, and profiles; Python tests cover interval accounting and capture warnings. No new end-to-end regression suite is included. Large planning-metadata accounting, a sustained slow trace writer, and worker-creation failure after callback enqueue lack coverage.
• Test results: tests were inspected only. No builds, tests, runtime stress, benchmarks, or format-mutating checks were run; no source files were modified. No regression .out files changed.
• Observability: shared counters and process metrics expose asynchronous tails, admission, fallback, and writeback. Trace checkpoints diagnose loss, but buffering is the new inline issue and packed lifecycle matching remains the existing thread.
• Transactions and persistence: no transaction protocol, EditLog, rowset-version selection, or delete-bitmap semantics changed; point reads retain rowset version/TSO context. Master failover logic is unaffected by this read/cache optimization.
• Data writes and crashes: cache writes are best effort, use immutable file keys, and check captured invalidation epochs. No additional stale authoritative-data write was established; the existing probe abort remains a BE availability problem.
• FE/BE variable propagation: not applicable; the feature is controlled by BE configuration and adds no transmitted variable.
• Performance and memory: payload allocations are tracked and have query/BE budgets, but retained page metadata and diagnostic strings escape that accounting as described inline. Repeated queued-fragment history remains separately covered by its existing thread. No additional verified performance issue emerged in the final recheck.

Coverage limits are static-only validation, absent end-to-end runtime/race/performance/ELF evidence, and the specific missing negative tests above. No unresolved static candidate remains in the accepted review scope.

Review converged after 3 round(s).

Existing P0/P1 findings confirmed for this head: #67292 (comment)

[github-actions]

[P2] Track the retained read-ahead page metadata

Enabling read-ahead copies every physical page in each scalar column into this plain std::vector, retained as ColumnReadAhead::_pages; dense scans also retain _candidate_pages and window-map nodes. These allocations have neither a tracked allocator nor manual MemTracker accounting, and happen before scheduler admission, so they survive even when every range is rejected. On 64-bit targets the two page arrays alone require at least 44 bytes per selected page: wide segments and concurrent scanners can retain substantial memory that query limits never see. Use tracked containers or balanced accounting/reservation for the planning metadata, preferably sharing immutable ordinal metadata, and cover the budget-rejection case.

[github-actions]

[P2] Bound the trace writer's buffered memory

With tracing enabled, a slow or stalled output disk lets all read-ahead and hole-fill producers keep appending here while the writer blocks in appendv() outside the mutex. _flush_bytes only wakes that writer; it does not cap either _pending or the in-flight batch, and both are untracked std::string allocations outside the read-ahead budgets. After a burst drains, batch.clear() still retains its capacity, including after tracing is disabled. Thus diagnostic capture can exhaust BE memory despite bounded query buffers. Account for and cap retained trace bytes, report dropped events through the existing checkpoints, and release excessive burst capacity.

[github-actions]

[P2] Preserve callback ownership after worker creation fails

With the supported segment_prefetch_thread_pool_thread_num_min=0, a transient failure creating the first worker can make ThreadPool::do_submit() return an error after it has already queued this callback; it does not roll that enqueue back. This branch removes the read from _inflight_reads and marks it FAILED. If the next range or batch successfully starts a worker, the retained callback runs and _mark_running() aborts because the state is no longer QUEUED. Dropping the scheduler can also leave that callback holding raw this. Ensure failed submission relinquishes the queued callback, or coordinate its execution/lifetime before finalizing the read. Cover one worker-creation failure followed by success, beyond the current capacity/shutdown rejection tests.

[shuke987]

run buildall

[bobhan1]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 28438 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8b140bad72379b6b153ddf1e5f235ae566a82c6, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17625	5156	5055	5055
q2	2147	325	260	260
q3	10198	1399	820	820
q4	4678	460	334	334
q5	7446	795	533	533
q6	178	180	151	151
q7	762	791	600	600
q8	9256	1375	1092	1092
q9	5451	4234	4191	4191
q10	6829	1283	1033	1033
q11	445	246	226	226
q12	630	396	275	275
q13	18124	3614	3068	3068
q14	276	269	252	252
q15	q16	738	708	657	657
q17	1201	751	570	570
q18	7087	6222	6113	6113
q19	1233	964	627	627
q20	376	343	229	229
q21	5046	2102	2274	2102
q22	357	301	250	250
Total cold run time: 100083 ms
Total hot run time: 28438 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5602	5564	5573	5564
q2	681	580	514	514
q3	5228	5366	4896	4896
q4	1979	2126	1533	1533
q5	5274	5181	5666	5181
q6	260	205	152	152
q7	2343	2136	1917	1917
q8	3021	2900	2674	2674
q9	8041	7877	8095	7877
q10	3778	3649	3226	3226
q11	625	447	392	392
q12	678	681	513	513
q13	3493	3926	3297	3297
q14	314	315	284	284
q15	q16	712	712	635	635
q17	8304	7494	7311	7311
q18	13078	12210	12856	12210
q19	907	799	774	774
q20	2162	2160	1948	1948
q21	5633	4695	4663	4663
q22	487	421	407	407
Total cold run time: 72600 ms
Total hot run time: 65968 ms

[hello-stephen]

TPC-DS: Total hot run time: 150596 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8b140bad72379b6b153ddf1e5f235ae566a82c6, data reload: false

query5	4315	583	443	443
query6	429	211	188	188
query7	4804	447	230	230
query8	321	164	151	151
query9	8774	3053	3064	3053
query10	477	309	255	255
query11	5894	3855	3545	3545
query12	146	90	86	86
query13	1251	445	347	347
query14	6103	3896	3615	3615
query14_1	3432	3360	3442	3360
query15	202	200	186	186
query16	933	416	405	405
query17	818	643	514	514
query18	2414	406	303	303
query19	197	158	128	128
query20	81	80	77	77
query21	205	123	102	102
query22	12881	13127	12758	12758
query23	13477	12885	12454	12454
query23_1	12604	12460	12416	12416
query24	7401	814	464	464
query24_1	456	454	464	454
query25	512	395	339	339
query26	1254	251	138	138
query27	2766	436	268	268
query28	4660	1938	1941	1938
query29	1547	582	428	428
query30	300	212	191	191
query31	1002	925	830	830
query32	164	92	93	92
query33	524	298	249	249
query34	977	842	501	501
query35	732	754	655	655
query36	826	834	728	728
query37	126	93	80	80
query38	1771	1719	1659	1659
query39	790	774	747	747
query39_1	735	732	729	729
query40	217	112	95	95
query41	70	62	65	62
query42	83	83	83	83
query43	337	364	308	308
query44	1298	685	685	685
query45	175	173	165	165
query46	814	948	554	554
query47	2904	2786	2847	2786
query48	289	301	222	222
query49	575	396	282	282
query50	666	283	212	212
query51	10797	10697	10553	10553
query52	75	78	74	74
query53	183	213	154	154
query54	235	191	177	177
query55	68	70	61	61
query56	259	220	209	209
query57	1649	1594	1582	1582
query58	282	260	242	242
query59	2070	2147	1969	1969
query60	271	242	227	227
query61	145	149	138	138
query62	404	331	281	281
query63	181	152	152	152
query64	2731	967	759	759
query65	3348	3266	3288	3266
query66	1767	428	298	298
query67	20042	20297	20037	20037
query68	3127	998	608	608
query69	421	279	245	245
query70	827	778	757	757
query71	299	234	228	228
query72	2700	2681	2343	2343
query73	510	532	290	290
query74	4667	4496	4365	4365
query75	2336	2300	1910	1910
query76	2391	1012	584	584
query77	337	361	269	269
query78	9722	9579	9233	9233
query79	873	836	510	510
query80	474	407	356	356
query81	528	323	264	264
query82	200	131	102	102
query83	192	195	197	195
query84	280	131	102	102
query85	768	422	377	377
query86	256	239	227	227
query87	1969	1988	1790	1790
query88	3580	2645	2635	2635
query89	305	277	242	242
query90	2102	162	165	162
query91	157	151	129	129
query92	93	81	83	81
query93	980	980	568	568
query94	464	309	277	277
query95	557	392	299	299
query96	664	490	229	229
query97	2445	2419	2328	2328
query98	158	150	144	144
query99	750	765	643	643
Total cold run time: 233033 ms
Total hot run time: 150596 ms

[hello-stephen]

ClickBench: Total hot run time: 24.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8b140bad72379b6b153ddf1e5f235ae566a82c6, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.11	0.10
query5	0.24	0.25	0.23
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.40	0.36	0.35
query10	0.57	0.56	0.59
query11	0.20	0.15	0.15
query12	0.19	0.14	0.14
query13	0.49	0.50	0.47
query14	1.03	1.01	1.03
query15	0.62	0.60	0.60
query16	0.34	0.35	0.31
query17	1.13	1.13	1.15
query18	0.21	0.20	0.17
query19	2.02	1.95	2.00
query20	0.02	0.01	0.01
query21	15.52	0.19	0.13
query22	4.97	0.06	0.05
query23	16.29	0.32	0.13
query24	2.55	0.69	0.40
query25	0.11	0.05	0.06
query26	1.52	0.21	0.15
query27	0.06	0.05	0.03
query28	4.11	0.77	0.36
query29	12.50	4.20	3.30
query30	0.28	0.14	0.10
query31	2.77	0.56	0.33
query32	3.22	0.60	0.50
query33	3.16	3.13	3.22
query34	15.32	4.00	3.42
query35	3.35	3.33	3.33
query36	0.56	0.45	0.43
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.16	0.15
query41	0.08	0.04	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.51 s
Total hot run time: 24.25 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 34.42% (53/154) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	66.97% (32375/48345)
Line Coverage	51.83% (342854/661441)
Region Coverage	47.21% (277774/588401)
Branch Coverage	48.89% (130962/267865)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 56.64% (81/143) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.43% (36270/46845)
Line Coverage	62.74% (410574/654402)
Region Coverage	59.14% (346723/586257)
Branch Coverage	60.07% (160503/267211)