[MOD-17189] Share SQ8 training state for RAM and disk indexes

[dor-forer]

Describe the changes in the pull request

RAM and disk tiered indexes need the same SQ8 training rules. Share the capped training threshold, double-precision sums, live-vector count, and mean calculation in SQ8TrainingState; RAM tiered HNSW uses this state while retaining its existing locks and jobs. Add the disk training-size parameter to the shared tiered configuration so the disk implementation can use the same state.

Which issues this PR fixes

Prerequisite for MOD-17189. Review and merge order: VecSim → RediSearch dependency/integration → RSE disk implementation. Downstream PRs pin this branch commit until it merges.

Main objects this PR modified

1. SQ8TrainingState: owns add/subtract/count/threshold/mean invariants.
2. RAM tiered HNSW: delegates training state; scheduling stays local.
3. TieredHNSWDiskParams: exposes the internal disk training-size setting; zero skips training.
4. Training tests: direct helper cases plus existing RAM SQ8 lifecycle coverage.

Validation

On dorer-intel, source commit 30983555573524436edc8c55960e18606965ad89 passed all 6 SQ8TrainingState component tests and all 162 HNSWTieredIndexTestSQ8*.* tests. The integrated RSE build using this commit passed 483 disk-vector tests and real Flex restart/training checks. Disk performance acceptance is pending in the downstream integration.

Mark if applicable

• This PR introduces API changes: an internal disk-tiered training-size parameter.
• This PR introduces serialization changes

---

Note

Medium Risk
Changes SQ8 training bookkeeping and finalize preconditions for tiered HNSW; behavior is intended to be equivalent but mistakes would affect quantization mean and backend initialization.

Overview
Introduces shared SQ8TrainingState for SQ8 training while vectors accumulate in the flat buffer: capped normalization threshold, FP64 running sums, live vector count, add/subtract, finalize check, and mean calculation.

Tiered RAM HNSW drops its nested SQAccumulationState and calculateQuantizationMean; it delegates to SQ8TrainingState and adds asserts that training count matches flat index size. Finalization now uses shouldFinalize() and calculateMean().

TieredHNSWDiskParams replaces the placeholder with QuantNormalizationSetSize (same semantics as tiered RAM HNSW) so disk tiered indexes can share the same training rules in follow-up work.

Adds component tests for SQ8TrainingState (float/float16, threshold cap, FP64 accumulation) and updates tiered SQ8 test accessors to use the new getters.

^{Reviewed by Cursor Bugbot for commit 3098355. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.57%. Comparing base (0348684) to head (3098355).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1078      +/-   ##
==========================================
- Coverage   97.58%   97.57%   -0.01%     
==========================================
  Files         142      143       +1     
  Lines        9444     9456      +12     
==========================================
+ Hits         9216     9227      +11     
- Misses        228      229       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.