fix(plan): make fulltext2 json_extract index probe reachable on current reads (#27926)

[cpegeric]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #27926

What this PR does / why we need it:

Makes a json_extract_string|json_extract_float64(col,'$.path') <op> const predicate
use a fulltext2 json-tuple index as a mandatory filter, correctly and without
dropping rows while the async (ISCP-maintained) index lags the base table.

The predicate is rewritten to:

base scan of t ──INNER JOIN on pk── candidate pks (from the probe)
(re-checks json_extract(...) on (fulltext2_search TVF, self-completing)
the row's CURRENT value at snapshot S)

The probe only needs to return a superset of matching pks — the base scan re-checks
the real predicate, so an extra candidate is dropped and (crucially) no matching row is
ever missed.

The problem

fulltext2 is always-async: an ISCP consumer tails the base table and appends index
segments, so the index trails the base by some lag. A naive index probe ANDed into a
predicate would silently drop rows committed inside that lag.

Design — self-completing probe with runtime tail binding

The fulltext2_search operator self-completes: after searching the bulk index it
runs the freshness-gap query itself and unions the results, so there is no plan-time
UNION ALL arm.

• bulk — rows the searched generation reflects (<= build_ts), matched through the index.
• tail — table_changes(db, t, searched, S] AS c WHERE c.change_type='insert' AND <json predicate>,
  projected to the pk, emitted as (doc_id=pk, score=0). The gap rows aren't in the index
  (that's the point), so the json predicate is evaluated directly on the changed rows —
  no index. S is the read point (current txn snapshot, or the {snapshot=...} TS).

A group-by dedup + INNER JOIN above narrow the union (bulk∩tail overlap on updates;
per-term repeats). Deletes need no handling — the tail is insert-only and the base fetch
is at S, so MVCC drops a pk deleted after build_ts.

Runtime tail binding (the correctness core)

The tail's lower bound is the generation the bulk search actually ran on, captured
atomically under the cache entry's read lock via the new RuntimeConfig.SearchedBuildTS
out-param — never read afterward. A post-search GetBuildTS would be a TOCTOU: the entry
lock releases when Search returns, so a concurrent evict+reload could publish a newer
generation and bind the tail above what was searched, dropping the rows in between.
Binding at search time closes that race with no plan-time generation pin.

Constraints honored

• table_changes runs inside the TVF (pkg/sql/colexec/table_function), never in
  pkg/fulltext2.
• The tail is streamed via RunStreamingSql, so a large gap streams in bounded batches
  rather than materializing every gap pk (OOM guard).
• The reconstructed tail SQL is surfaced on the scan node's Stats.Sql, so
  EXPLAIN (VERBOSE) shows it (same mechanism as classic fulltext) — the internally-run
  tail is visible, not a black box. It is shown only when the tail is expected to run
  (index behind as of planning); a caught-up probe shows none.

Coverage gate

The plan decides probe vs full scan (not covered-vs-behind, since self-completion
makes them equivalent). It probes when the index is usable — built (build_ts > 0),
table_changes can serve the table (ordinary, explicit non-hidden pk), and there are no
transaction-local writes to the source (those aren't visible to the index or the tail →
full scan). Fails closed on any uncertainty.

Current and {snapshot=...} reads share one path: a snapshot read binds the
snapshot-keyed generation, the snapshot TS as the tail's upper bound, and the snapshot's
owning account.

Key files

File	Change
pkg/sql/plan/apply_indices_fulltext_json.go	decideJSONProbe (probe/skip); recordJSONProbeTail +
jsonComparisonSQL/jsonLiteralToSQL (rebuild the predicate as SQL over the source column); gated display SQL
pkg/sql/plan/apply_indices_fulltext.go	splice: single self-completing node (no UNION), publish Stats.Sql
pkg/sql/plan/apply_indices_fulltext2.go	buildFulltext2SearchCfg emits probe_tail/src/pkey/probe_tail_where
pkg/sql/colexec/table_function/fulltext2_search.go	streaming tail
(startProbeTail/emitProbeTail/appendTailResult/closeProbeTail), SearchedBuildTS binding, aliased table_changes
pkg/vectorindex/types.go, pkg/vectorindex/cache/cache.go	RuntimeConfig.SearchedBuildTS out-param, populated under the
entry lock
pkg/fulltext2/{storage.go,search_cache.go,plugin/coverage/coverage.go}	TableConfig.ProbeTail/ProbeTailWhere; removed
the maxTs pin; searchedBuildTS warm-or-cold
docs/design/json_tuple_wordbreaker.md	§10 aligned to the implemented semantics

Testing

• Unit (pkg/sql/plan, pkg/sql/colexec/table_function, pkg/fulltext2/...,
  pkg/vectorindex/cache): probe decision matrix, tail-SQL reconstruction, streaming
  tail emit/close, empty-gap skip. -race clean.
• BVT (pessimistic_transaction/fulltext2, clean instance, -g -n): json-probe
  154/154 + datalink 49/49 = 203/203, 100% — covering current/snapshot, covered/partial,
  fallback (txn-local write → full scan), composite-pk decline, ALTER, prepared, multi-col.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[cpegeric]

bug fixed.

[XuPeng-SH]

Re-reviewed exact head e3db4a7 against main 9a5dd0c. REQUEST_CHANGES: the previous execution-generation completeness requirement is improved but not closed.

[P1] A planning-time GetBuildTS observation does not bind the generation used by the mandatory probe.

Concrete interleaving:

1. On CN B, an earlier query at snapshot S0 starts loading G0 under the ordinary index-table cache key; pause that loader before STATUS_LOADED.
2. On CN A, commit a matching JSON row at T > S0 and let ISCP persist G1/build_ts W1 >= T. CN B observes the source commit and plans a new read S1.
3. GetBuildTS returns found=false for B's in-flight entry. searchedBuildTS consequently reads durable W1 and declares the new query covered, emitting no gap tail.
4. Resume B's loader. The new query's Cache.Search uses LoadOrStore on the same key, waits for the existing entry, and searches G0 loaded with the first caller's snapshot. There is no required-build-ts/snapshot validation at search time. G0 omits the new PK, and the mandatory join drops a row visible at S1.

This is not merely an atomic-read race: the atomic protects the field, not the lifetime/identity of the chosen generation. Even a loaded entry can be replaced between planning and execution. A generation refreshed after a current query's snapshot can also delete postings that its base snapshot still needs; a lower-bound watermark alone cannot establish snapshot compatibility.

Please pin/bind a compatible execution generation or make the exact JSON-probe execution path independently enforce the coverage/snapshot requirement with a correctness-preserving fallback/tail. Treating in-flight entries as genuinely absent is immediately unsafe; changing that case alone does not solve replacement after the check. The partial tail also needs its lower bound tied to the generation actually searched. Ordinary MATCH's eventual-consistency contract can remain unchanged. Add a barrier-driven loader/planner/search regression for this interleaving and a refresh-after-read-snapshot deletion case.

The previous fixed-delay issue has been removed. The new TN-object commit-zonemap handling, transaction-local/partitioned fail-closed guards, partial UNION ALL + PK dedup with the original base predicates retained, and prepared-plan rebuild are substantive improvements; this review does not repeat the removed delay arithmetic finding.

The design record still needs alignment before approval: section 10.2 says historical reads compare directly with S without SourceCommitTS; section 10.3 says snapshots never use partial plans and promises a small-gap cost gate/DDL fallback. This head explicitly uses historical SourceCommitTS and historical partial plans with no cost gate. Update the versioned contract together with execution-generation binding; the PR body and several comments also disagree with that implementation.

Validation: inspected the current production change map, prior review findings, coverage/cache/search ownership paths, planner/tail/prepare changes, source-commit persistence transitions, and focused test/BVT oracles. The new GetBuildTS unit test explicitly expects an in-flight entry to return not-found, but does not compose that result with durable fallback and the waiting Search consumer. Author-reported UT/BVT/GPU results were considered as reported; I did not run a fresh native suite, GPU workload or live two-CN reproducer and did not wait for CI. No repository files were modified; head/base were rechecked before posting.

[jiangxinmeng1]

Reviewed the readiness calculation in the latest commits.

The async fulltext2 probe is now gated by:

watermark >= SourceCommitTS

instead of comparing the watermark directly with the wall-clock SnapshotTS.

SourceCommitTS is evaluated at the transaction snapshot and covers:

• in-memory source DML commit timestamps with commit_ts <= snapshot;

• appendable-object commit timestamps with commit_ts <= snapshot;

• CN-created objects through NewObjectsIter(snapshot, onlyVisible=true), which applies CreateTime <=
  snapshot and the corresponding delete visibility check;

• the partition-state retention boundary.

The coverage hook uses wm.LT(req.SourceCommitTS) as the fail-closed condition, which is equivalent
to requiring watermark >= SourceCommitTS.

I do not see a correctness issue in the current readiness calculation.

[aptend]

Re-review of exact head e3db4a7 against base 9a5dd0c. I read the complete review/comment/thread history, checked the delta since my prior reviewed head b9644e0, and re-audited the full 59-file diff. The earlier fixed-delay and flushed-object coverage gaps are materially improved by BuildTS, SourceCommitTS, the partial tail, and prepared-plan rebuilding. Two blockers remain:

1. The planning decision still does not bind the index generation later searched. A loading cache entry is treated as a cache miss here, so planning may use a newer durable build_ts while execution waits for and searches that older in-flight entry. The resulting INNER JOIN can drop committed matching rows; the partial plan can also begin its tail after a gap that the executed generation does not cover.
2. The mandatory design record is not exact for the implementation under review: it says historical reads use the snapshot as the bar and never use partial coverage, and requires a small-gap/DDL cost gate, while the current code/tests use SourceCommitTS plus partial plans for snapshots and emit partial plans without that gate. This high-risk planner/cache/MVCC change needs the exact implemented contract reviewed and approved before merge.

Validation: exact-head CI is green and git diff --check passes. I did not rerun native/GPU/cluster suites locally because the generation race is source-provable and the design gate already fails.

[aunjgr]

Re-reviewed exact head e3db4a7 against base 9a5dd0c. Two blockers remain.

[P1] Coverage must bind the generation execution actually searches, including in-flight loads.

pkg/vectorindex/cache/cache.go:1136-1148 returns found=false for an existing entry that is not STATUS_LOADED. searchedBuildTS (pkg/fulltext2/plugin/coverage/coverage.go:183-198) treats that as a cold miss and uses MAX(metadata.build_ts). But Cache.Search/SearchInto uses LoadOrStore on the same key and waits for that existing entry, rather than necessarily loading the generation planning measured.

Concrete schedule: query A begins loading generation W0; a matching row commits at T>W0 and ISCP publishes W1>=T; query B plans while A's entry is still loading. B's gate reads durable W1 and accepts coverage. A finishes W0, and B executes against that older entry. The mandatory join drops the matching row. In the partial variant, starting table_changes after W1 also leaves the missing interval (W0,W1] uncovered. Retaining the base predicate cannot recover a missing candidate.

Bind/retain a snapshot-compatible generation or enforce its coverage at execution with a safe fallback. Distinguishing absent from loading is necessary but does not by itself solve replacement between planning and search or execution on another CN. Add deterministic held-load and plan-to-search generation tests.

Design gate: the expanded partial-probe feature crosses planner, MVCC, cache generations and table_changes. docs/design/json_tuple_wordbreaker.md:535-565 says historical reads use snapshot TS, never partial coverage, and partial plans require a small-gap/DDL gate. The implementation uses SourceCommitTS for historical reads and permits historical partial plans; decideJSONProbe:426-435 explicitly has no cost gate. The PR body also describes a different historical contract. Align the exact versioned design with the chosen behavior and obtain review of that revision.

The earlier warm-loaded-cache coverage problem is materially improved, but this loading-generation path remains. I independently traced the source path, confirming the existing exact-head review. This is a design-gated incremental review, not a claim of a fresh full 59-file implementation approval. Reused green exact-head CI; no new native/GPU/cluster run.

[cpegeric]

fixed and add schema version check