feat: support Nemotron-H / Nemotron-Cascade-2 (#1711)

[michael-rabe]

Adds initial support for hybrid Mamba2 + Attention + MoE models via the unfused_moe registry. A bare AutoRound("nvidia/...") call now produces a coherent INT4 checkpoint without launcher-side workarounds.

Core additions:

• unfused_moe/nemotron_h.py: linear-discoverable MoE block
• unfused_moe/nemotron_h_setup.py: post-load fixups (Zamba2 group_size, SSM/router FP32 restore)
• utils/source_tensor_overrides.py: generic source-checkpoint tensor reload utility
• MODEL_CONFIG["nemotron_h"]: dispatch + upstream-rename preservation
• compressors/base.py: apply_post_load_fixups hook (non-NH no-op)
• Export: norm_dtype and scale_dtype per-layer overrides

Documentation:

• New .claude/skills/adapt-unfused-moe skill covering the pipeline
• adapt-new-llm slimmed to point at the dedicated skill
• README / README_CN / step_by_step notes updated

Tests: 89 CPU tests across registration, post-load, export dtype controls, source-tensor overrides, and missing-tensors symmetry.

Closes #1711

Description

Please briefly describe your main changes, the motivation.

Type of Change

• Bug fix
• New feature
• Documentation update
• Performance improvement
• Code refactoring
• Other (please specify):

Related Issues

Fixes or relates to #

Checklist Before Submitting

• My code has been tested locally.
• Documentation has been updated as needed.
• New or updated tests are included where applicable.

[wenhuach21]

Thanks for the PR. Are all these file changes necessary? It seems to modify many unrelated files.

[wenhuach21]

For this case, we should provide a better solution, similar to e_score_correction_bias's handling in deepseek

@xin3he is it a regression as I just found the dtype of https://huggingface.co/Intel/DeepSeek-V3.2-int4-AutoRound/blob/main/model-00059-of-00071.safetensors if bf16

[michael-rabe]

Thanks for the PR. Are all these file changes necessary? It seems to modify many unrelated files.

I'm afraid it is, but haven't optimized (reduced) much after I got coherent and stable output.
Looking forward to your feedback ;).

Very welcome if we can reduce the number of changes, nemotron models seem to behave way different to others and are more sensitive to low dtypes.
I had coherence problems without these changes, the post apply "patches" became necessary to bypass the "standard" autoround behaviours without changing too much of the autoround core.

[wenhuach21]

Thanks for the PR. Are all these file changes necessary? It seems to modify many unrelated files.

I'm afraid it is, but haven't optimized (reduced) much after I got coherent and stable output. Looking forward to your feedback ;).

Very welcome if we can reduce the number of changes, nemotron models seem to behave way different to others and are more sensitive to low dtypes. I had coherence problems without these changes, the post apply "patches" became necessary to bypass the "standard" autoround behaviours without changing too much of the autoround core.

Thanks for the prompt reply. Could you let me know what indicator led you to change the norm dtype to FP32? I checked the model and noticed that the original norm weights are in BF16. I’m wondering whether modifying the norm dtype in the weights might cause issues with vLLM or SGLang, and whether they would still behave as expected.

For MoE models, I would generally suggest using higher precision for certain parts, for example, 8-bit for critical components (non-moe modules) and 4-bit for others (experts). This is a more common approach when pure 4-bit quantization leads to a significant accuracy drop and avoids the API change.

Please feel free to correct me if I’m mistaken, as I’m not very familiar with this model.

Thanks again for your pr.

[wenhuach21]

for mixed-bits, you could try scheme="int4_mixed"

[michael-rabe]

Thanks for the PR. Are all these file changes necessary? It seems to modify many unrelated files.

I'm afraid it is, but haven't optimized (reduced) much after I got coherent and stable output. Looking forward to your feedback ;).
Very welcome if we can reduce the number of changes, nemotron models seem to behave way different to others and are more sensitive to low dtypes. I had coherence problems without these changes, the post apply "patches" became necessary to bypass the "standard" autoround behaviours without changing too much of the autoround core.

Thanks for the prompt reply. Could you let me know what indicator led you to change the norm dtype to FP32? I checked the model and noticed that the original norm weights are in BF16. I’m wondering whether modifying the norm dtype in the weights might cause issues with vLLM or SGLang, and whether they would still behave as expected.

For MoE models, I would generally suggest using higher precision for certain parts, for example, 8-bit for critical components (non-moe modules) and 4-bit for others (experts). This is a more common approach when pure 4-bit quantization leads to a significant accuracy drop and avoids the API change.

Please feel free to correct me if I’m mistaken, as I’m not very familiar with this model.

Thanks again for your pr.

Ah, I missed the proper documentation (I'll fix that tonight).
Thanks for the valuable suggestions, I'm following up on that and set the PR into "draft" status.

You're right that the original norm weights are BF16, and BF16 is numerically fine for the norm weights themselves — upcasting them to FP32 is lossless but gains nothing at the weight level.

The purpose of norm_dtype="fp32" is different: it's a lever for the residual stream, not for the norm weights. In HuggingFace-style RMSNorm the norm output's dtype follows the weight's dtype, so storing norm weights as FP32 pulls the post-norm tensor — and via the residual add, the residual stream itself — into FP32 at runtime. Over 50+ layers the residual is a sum of many block outputs, and keeping that accumulation in FP32 reduces BF16 rounding drift.

Two caveats: this only helps on engines that honor the stored norm-weight dtype (HF transformers does, fused-kernel runtimes like vLLM/SGLang often don't). And the Nemotron-H coherence fix itself was not norm_dtype — it was the always-on post-load restore of the SSM core tensors and the router correction bias to FP32, which are the tensors with genuine FP32 requirements. norm_dtype is orthogonal and optional.

So: FP32 norm weights aren't about upgrading the norm weights themselves — they're an export-side lever for residual-stream precision on compliant inference stacks.

Please correct me, if I'm wrong. I'm new to the club and still in the learning phase ;).
I appreciate your questions.
Sorry for the wall of text.

[michael-rabe]

Progressing, the comments were helpful and can reduce the number of changes.
Still investigating as I'm not yet happy with the results (accuracy) and want to make sure it's not in the quantisation itself.

[michael-rabe]

I tried to remove everything not absolutely necessary (some fallbacks added), I ran a couple of autoround runs to verify the results (BF16 wasn't the actual issue, ignore that comment, the differences are neglectable).
Test-Results: LAMBADA-5153: acc=0.624, tok_acc=0.704.