fix(query): close the two ways an unresolved name still passed a select - #437
Conversation
The general case was fixed with #414 — an unknown column reference gets an invalid-scan marker that ray_execute rejects. Two ways around it survived, and both reproduced #428 in full: a projection emitted under the wrong label, and an aggregate answering its neutral element. `_dist` was exempt from the marker unconditionally. The exemption is needed — a rerank produces that column while the query runs, so it is genuinely absent when the query compiles — but it applied just as well where no rerank exists to produce it. It is now conditional on the graph actually holding one. `nearest:` was missing from the skip list in the loop that names output columns. That walk is positional, so a keyword it fails to skip eats an output slot and every name after it lands one column early. It is counted out where the outputs are counted, which is why only the naming disagreed. Both were verified to reproduce the report before the change: the projection case returned a two-column table whose second column carried the third column's data under the second column's name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
singaraiona
left a comment
There was a problem hiding this comment.
Reviewed with an empirical verification pass (head + base built ASan/UBSan, repros run on both). The fix is correct and does exactly what it says — both escape routes for an unresolved name are closed: the _dist exemption is now gated on a rerank node actually being present, and nearest_id is added to the output-naming skip list so it matches the output-count loop and select_output_count exactly. Suite 3713/3713 clean. Approving.
Verified, so reviewers don't re-chase them:
where: _distis not a regression.(select {i: id from: TN where: (< _dist 0.5) nearest: (knn Vec [1.0 0.0]) take: 2})fails with a schema error on both head and base — the WHERE reference compiles before the rerank node exists, so the exemption legitimately can't fire there. Head just gives a more specific message. Not newly broken.- The keyword sets now agree exactly, the keyword universe is closed at seven, and every other positional dict-walk that still omits
nearest_id(7056/7341/7614/8147/8575/8818/2480/2561) is inside aby:-only path, unreachable today becausenearest:+by:is rejected up front at query.c:5727.
One optional hardening (non-blocking): the regression test's [1 3] assertion rests on an exact floating-point tie — id 3 [1.0 1.0] and id 4 [0.5 0.5] have identical cosine distance to q=[1,0], and the result is decided only by the heap keeping the earlier-inserted row. On this platform the two distances are bit-identical at -O0 and -O2 (both reduce to correctly-rounded sqrt(0.5), so it's actually not FMA/libm-dependent for these inputs), so it passes robustly — but the test asserts a golden that rests on a tie rather than a real ordering, which is needlessly fragile for the next contributor. Changing id 4's vector to make the distances distinct (e.g. [0.5 0.5] → [0.9 0.1]) exercises the identical path and passes with zero golden edits — verified.
Two pre-existing follow-ups surfaced while tracing (neither introduced by this PR, both behave identically on base — worth a separate issue, not a blocker here):
- The projection eval-fallback silently drops
nearest:when there's nowhere::(select {u: (distinct id) from: TN nearest: (knn Vec [1.0 0.0]) take: 2})returns table-order rows, not nearest, with no error (confirmed same on head and base).distinctdoesn't DAG-lower, so the fallback frees the graph including the rerank. - The exemption keys on graph state rather than the query — it works because outputs compile after the rerank node, but that coupling to compile order isn't enforced by anything.
nearest_expr != NULL(already computed at query.c:5690) would be order-independent; worth considering if this area is touched again.
Approving and merging — I'll fold the test-tie tweak suggestion in as a comment rather than blocking on it.
Fixes #428
Already fixed, except for two ways around it
The report was filed against
v2.5.15and describes the same defect as #414, which has since been fixed: an unknown column reference now gets an invalid-scan marker thatray_executerejects. Every shape in the report raisesschema: column 'nosuch' not foundon currentdev.Two ways around that marker survived, and both reproduce the report in full — a projection emitted under the wrong label, and an aggregate answering its neutral element. Neither should close as a duplicate.
Hole A — the
_distexemption was unconditional_distis exempt from the marker because a rerank produces that column while the query runs, so it is genuinely absent when the query compiles. But the exemption applied just as well to a query with nonearest:clause, where nothing will ever produce it. Verified ondevbefore the change:The exemption is now conditional on the graph actually holding an
OP_ANN_RERANK/OP_KNN_RERANKnode. The rerank is built before projections compile, so the check is a walk over nodes already present, on the cold unknown-name path only.Hole B —
nearest:missing from the rename loop's skip listThe loop that names output columns is positional, so a clause keyword it fails to skip consumes an output slot and every name after it lands one column early.
nearest:is counted out where the outputs are counted but was not skipped here, so a query naming it before its outputs labelled them one position off:Not changed
The marker-plus-
validate_scan_columnssplit stays as it is. Raising at resolution time was considered and rejected for the reason the existing comment records: optimizer rewrites can legitimately make a scan dead, and an eager error would have to be threaded through everycompile_expr_dagcaller's NULL contract.Tests
test/rfl/regress/issue_428.rflcovers both holes and the legitimate_distcase that must keep working. Confirmed red before the change: the projection shape returned a two-column table whose second column carried the third column's data under the second column's name.No existing test depended on the silent behaviour — the assertions in this area all demand the error.
Suite: 3713/3713.
🤖 Generated with Claude Code