[MOD-17916] Bound native FP16 dispatch without hot-path overhead

[dor-forer]

Current stack: #1018 → #1019 → #1021.

#1021 is based directly on #1019 commit 809b522b.

Describe the changes

This PR adds conservative dimension guardrails around native-FP16 distance functions without
adding work to their hot loops.

• Native-FP16 inner product is selected through 65,504 dimensions.
• Native-FP16 L2 is selected through 16,376 dimensions.
• These thresholds use unit-scale worst cases (1 per IP component and 4 per L2 component) as a
  conservative dispatch heuristic, not as an input-range contract.
• Above the caps, dispatch falls through to the existing FP32-accumulating implementations.
• The AVX512FP16 inner-product tail subtracts the reduced dot product from 1.0f in FP32, avoiding
  an additional half-precision rounding near 1.

The cap is checked once when the distance function is selected. AVX512FP16, NEON_HP, SVE, and SVE2
kernels below the cap retain their native-FP16 accumulation and reduction loops, with no added
branch, conversion, or fallback cost per distance calculation.

Correctness boundary

IP and L2 inputs are not required to be in [-1, 1]. There is no universally safe positive
dimension for arbitrary native-FP16 arithmetic: a single product or squared difference can
overflow. These caps are guardrails against dimension-driven overflow for unit-scale data; they
are not a general overflow guarantee, and sufficiently large values can overflow below the caps.

Native-FP16 arithmetic is also approximate by design. Subtraction, lane FMAs, and final reduction
can round ordinary finite L2 scores differently from scalar FP32 accumulation. FLOAT16 L2 flow tests
therefore validate the existing 1% native-kernel accuracy contract, including score ties and labels
near a KNN or range boundary, rather than requiring bitwise-equivalent rankings across runner CPUs.
Other data types and metrics retain their existing strict checks.

Performance decision

• Native FP16 IP and L2 dispatch remains enabled on x86 and ARM below the caps.
• Normal embedding dimensions keep the pre-PR native hot path.
• High dimensions use an existing wider-accumulator tier, where avoiding dimension-driven overflow
  is worth the implementation-tier change.
• Unconditional FP32 final reduction and conditional recomputation were rejected because they add
  common-path cost and do not repair rounding that already occurred in native-FP16 lane arithmetic.

Tests

• Pin native dispatch at the cap and wider dispatch immediately above it.
• Compare advanced native-FP16 tests with scalar FP32 references over the stored FP16 values, using
  the documented 1% tolerance.
• Check the unit-scale high-dimension guardrail through public dispatch.
• Check the AVX512FP16 inner-product subtraction is evaluated in FP32.
• Validate FLOAT16 L2 KNN, range, batch-iterator, and multivalue behavior within the native-FP16
  accuracy contract.

Local verification:

• GCC 13 release build of test_spaces passed.
• test_spaces: 1,587 tests passed (one hardware-dependent case skipped locally).
• The six targeted FLOAT16 L2 flow tests passed.
• The previously failing Intel AVX512FP16 KNN sample satisfies the 1% score/ranking contract.
• Python syntax checks, repository format check, and git diff --check passed.
• AArch64 GCC 13 and Clang 18 cross-compilation previously passed with NEON_HP, SVE, and SVE2
  enabled.

Which issue this PR fixes

• MOD-17916

Related performance work: MOD-16688 / PR #984.

Files modified

1. src/VecSim/spaces/spaces.h
2. src/VecSim/spaces/IP_space.cpp
3. src/VecSim/spaces/L2_space.cpp
4. src/VecSim/spaces/IP/IP_AVX512FP16_VL_FP16.h
5. tests/unit/test_spaces.cpp
6. tests/flow/common.py
7. tests/flow/test_bruteforce.py
8. tests/flow/test_hnsw.py

Mark if applicable

• This PR introduces API changes
• This PR introduces serialization changes

---

Note

Medium Risk
Changes which distance kernel runs for large FP16 dimensions and alters AVX512 FP16 IP numerics near 1.0; behavior is guarded and heavily tested but affects core search scoring.

Overview
Adds one-time dimension caps so native half-precision IP/L2 SIMD is only chosen below conservative limits (IP through 65,504, L2 through 16,376); larger dimensions fall back to existing wider-accumulator/scalar paths with no per-distance overhead.

On AVX512 FP16+VL inner product, the kernel now uses dual FMA accumulators and a 64-element specialization loop, and computes 1.0f − dot in FP32 after reduction to avoid extra rounding near distance 1.

Tests pin dispatch at/above the caps, verify high-dimension overflow guardrails, and relax FLOAT16 L2 flow/HNSW checks to the documented 1% native-kernel tolerance (including ties and range boundaries) instead of exact FP32 equality.

^{Reviewed by Cursor Bugbot for commit 92452fb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.28%. Comparing base (2546f8e) to head (92452fb).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1021   +/-   ##
=======================================
  Coverage   97.28%   97.28%           
=======================================
  Files         141      141           
  Lines        8624     8630    +6     
=======================================
+ Hits         8390     8396    +6     
  Misses        234      234           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c1ef771. Configure here.}

[cursor]

KNN tolerance bands ignore two-sided error

Low Severity

assert_float16_l2_knn builds mandatory and allowed with one-sided factors 1 ± rtol around the exact k-th distance. KNN ranking compares two noisy scores, so both the candidate and the cutoff neighbor can each move by rtol. Guaranteed members need a tighter band near (1 - rtol) / (1 + rtol), and the allowed set needs a wider one near (1 + rtol) / (1 - rtol). As written, the helper can fail on label sets that still satisfy the 1% native-FP16 contract.

 

^{Reviewed by Cursor Bugbot for commit c1ef771. Configure here.}