Skip to content

fix(schedulers): promote DEIS polynomial coefficients to float64 to prevent catastrophic cancellation - #15159

Open
MinYeFa wants to merge 1 commit into
huggingface:mainfrom
MinYeFa:fix/deis-scheduler-float64-coefficients-15090
Open

MinYeFa wants to merge 1 commit into
huggingface:mainfrom
MinYeFa:fix/deis-scheduler-float64-coefficients-15090

Conversation

@MinYeFa

@MinYeFa MinYeFa commented Oct 30, 2026

Copy link
Copy Markdown

Summary

Fixes #15090.

In DEISMultistepScheduler, evaluating polynomial integral coefficients in float32 leads to catastrophic cancellation as num_inference_steps increases (when the step size $h = \Delta \log\rho \ll 1$).

Specifically, in multistep_deis_second_order_update and multistep_deis_third_order_update:

  • rho_t, rho_s0, rho_s1, rho_s2 were calculated via PyTorch float32 division.
  • Passing float32 tensors into np.log produces single-precision NumPy floats.
  • In ind_fn, the polynomial integration formulas compute finite differences between adjacent terms whose difference is $\mathcal{O}(h^2)$ or $\mathcal{O}(h^3)$ while individual terms are $\mathcal{O}(1)$.
  • For 500 inference steps ($h \approx 0.002$), the denominator is $\sim 10^{-6}$ and floating-point cancellation in float32 exhausts all 24 bits of mantissa precision, leading to >8% relative trajectory error.

Solution

  • Promotes rho_t, rho_s0, rho_s1, rho_s2 to standard 64-bit Python floats (float(sigma / alpha)).
  • Evaluates the polynomial coefficients in full 64-bit float precision before scaling model_output_list tensors (float(...)).
  • Added a dedicated regression test test_high_step_coefficient_precision verifying stable ODE integration across 50 and 100 inference steps for orders 2 and 3.

…revent catastrophic cancellation (huggingface#15090)

In DEISMultistepScheduler, evaluating polynomial integral coefficients in float32
leads to catastrophic cancellation when num_inference_steps is large (h << 1).
For 500 steps, float32 precision loss exceeds 8%, whereas evaluating the polynomial
integrals in float64 maintains numerical stability across all step regimes.

This commit converts the intermediate log-rho values and integral evaluations in
multistep_deis_second_order_update and multistep_deis_third_order_update to float64
scalars before scaling the model output tensors.

Signed-off-by: Min-Ye Fa <347497424+MinYeFa@users.noreply.github.com>
@MinYeFa
MinYeFa force-pushed the fix/deis-scheduler-float64-coefficients-15090 branch from e94fa86 to 86e2f28 Compare October 31, 2026 22:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DEIS coefficient precision causes increasing error as inference steps increase

1 participant