Fix/dsv4 native transformers warmup by grimoire · Pull Request #4878 · InternLM/lmdeploy

grimoire · 2026-08-18T09:21:02Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

Copilot

Pull request overview

This PR updates LMDeploy’s DeepSeek-V4 integration to align with native Transformers config/schema and improves the CUDA V4 indexer warmup and metadata handling, while removing LMDeploy’s custom HF config registration paths.

Changes:

Switch DeepSeek-V4 compression config handling to native layer_types/compress_rates and force V4 cache block sizing to a fixed 256.
Extend the V4 indexer backend interface to include num_heads/head_dim, and fix DeepGEMM warmup + empty-sequence handling via new topk_seqlens.
Remove LMDeploy custom DeepSeek (v4/v32) HF config classes and manual config registration/export hooks.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/pytorch/config/test_model_config.py	Updates tests to reflect native V4 layer schema and fixed block sizing expectations.
lmdeploy/pytorch/transformers/configuration_deepseek_v4.py	Removes re-export shim for the custom DeepSeek-V4 HF config.
lmdeploy/pytorch/transformers/configuration_deepseek_v32.py	Removes re-export shim for the custom DeepSeek-V32 HF config.
lmdeploy/pytorch/transformers/init.py	Drops `register_config` from the public pytorch transformers helper exports.
lmdeploy/pytorch/nn/v4_indexer.py	Updates V4 indexer wrapper to pass `num_heads`/`head_dim` into backend builder.
lmdeploy/pytorch/models/deepseek_v4.py	Adapts DeepSeek-V4 model wiring to native config fields and new compression-ratio translation.
lmdeploy/pytorch/configurations/deepseek_v4.py	Adds native layer schema translation, forces `V4_BLOCK_SIZE=256`, updates cache config finalization.
lmdeploy/pytorch/backends/indexer.py	Extends `BaseV4IndexerBuilder.build()` signature to include `num_heads`/`head_dim`.
lmdeploy/pytorch/backends/cuda/v4_indexer.py	Fixes DeepGEMM warmup shapes/keys and requires `topk_seqlens` for packed scoring path.
lmdeploy/pytorch/backends/cuda/attention/v4.py	Adds `topk_seqlens` to index-score metadata and clamps scheduler inputs safely for empty rows.
lmdeploy/hf_configs/configuration_deepseek_v4.py	Deletes LMDeploy’s custom DeepSeek-V4 `PretrainedConfig` implementation.
lmdeploy/hf_configs/configuration_deepseek_v32.py	Deletes LMDeploy’s custom DeepSeek-V32 `PretrainedConfig` implementation.
lmdeploy/hf_configs/init.py	Simplifies config loading to a direct `AutoConfig.from_pretrained()` call.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

 from transformers import AutoConfig

-from lmdeploy.utils import get_logger
-
-logger = get_logger('lmdeploy')
-
-
-@lru_cache
-def register_config(model_type: str):
-    if model_type == 'deepseek_v32':
-        from .configuration_deepseek_v32 import DeepseekV32Config
-        AutoConfig.register(DeepseekV32Config.model_type, DeepseekV32Config)
-    elif model_type == 'deepseek_v4':
-        from .configuration_deepseek_v4 import DeepseekV4Config
-        AutoConfig.register(DeepseekV4Config.model_type, DeepseekV4Config)
-    else:
-        logger.debug(f'Can not register config for model_type: {model_type}')
-

 def config_from_pretrained(pretrained_model_name_or_path: str, **kwargs):
-    try:
-        return AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
-    except Exception as e:
-        logger.debug(f'AutoConfig.from_pretrained failed: {e}, try register config manually.')
-        # some models do not provide auto map for config
-        from transformers import PretrainedConfig
-        trust_remote_code = kwargs.pop('trust_remote_code', None)
-        config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
-        model_type = config_dict.get('model_type', None)
-        if trust_remote_code is not None:
-            kwargs['trust_remote_code'] = trust_remote_code
-        register_config(model_type)
-        try:
-            return AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
-        except Exception as e:
-            return PretrainedConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
+    return AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)


+    compress_rates = hf_config.compress_rates
+    compressed_layer_types = set(layer_types).difference({'sliding_attention'})
+    missing_rates = sorted(compressed_layer_types.difference(compress_rates))
+    if missing_rates:
+        raise ValueError(f'DeepSeek-V4 compress_rates is missing layer types: {missing_rates}.')
+
+    compress_ratios = [
+        0 if layer_type == 'sliding_attention' else compress_rates[layer_type]
+        for layer_type in layer_types
+    ]


lvhan028 · 2026-08-24T04:16:50Z

May put "pip install tile-kernels" in docker/install.sh

lvhan028 · 2026-08-24T04:46:01Z

{
  "model": "/mnt/shared-storage-gpfs2/gpfs2-shared-public/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731/snapshots/7872f01b1d1fe23eabc4c98b48bffcef5a386062/",
  "messages": [
    {
      "role": "user",
      "content": "What is the weather like in San Francisco, CA? Use the weather tool."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string",
              "description": "The city to find the weather for, e.g. San Francisco"
            },
            "state": {
              "type": "string",
              "description": "The state abbreviation, e.g. CA"
            },
            "unit": {
              "type": "string",
              "description": "The unit for temperature",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["city", "state"]
        }
      }
    }
  ],
  "tool_choice": "none",
  "temperature": 0,
  "max_completion_tokens": 1024
}

The response log shows:

2026-08-24 04:40:42,982 - lmdeploy - REQUEST - logger.py:55 - session=0, response='\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke>\n<｜DSML｜invoke name="get_current_weather">\n<｜DSML｜parameter name="city" string="true">San Francisco</｜DSML｜parameter>\n<｜DSML｜parameter parameter="state"="CA">CA</｜DSML｜parameter>\n<parameter name="unit">fahrenheit</parameter>\n</｜DSML｜parameter>\n</｜DSML｜inv>\n</｜DSML｜invoke>\n</｜DSML｜tool>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>'

This does not appear to be a valid tool_call response. Could you please help investigate whether this is caused by the model, the chat template, the inference pipeline, or something else?

Co-Authored-By: Claude <noreply@anthropic.com>

tile-kernels depends on a cu13 tilelang build, so gate the pip install on CUDA_VERSION_SHORT == cu13* instead of installing it unconditionally. Co-Authored-By: Claude <noreply@anthropic.com>

The _hc_post_expand_kernel received *comb.stride() for a contiguous [n, src, out] tensor, assigning comb_stride_out_h=stride(1) (the SRC axis) and comb_stride_src_h=stride(2) (the OUT axis). The kernel indexes weight as (out_h, src_h) and sums over src_h, so this loaded comb[out, src] and computed matmul(comb, residual) -- the transpose of the intended residual mix. DeepSeek-V4's comb is produced by hc_split_sinkhorn as [n, src, out], identical to native transformers / official inference / vllm, all of which apply matmul(comb.T, residual) == sum_src comb[src, out]*residual[src]. The Sinkhorn comb is doubly-stochastic but non-symmetric (real weights ~2-11% off-diagonal), so the transposed application accumulated systematic error across 86 applications (2 per layer x 43 layers) and flipped low-margin closing-tag tokens in DSML tool-call output, e.g. </｜DSML｜tool> instead of </｜DSML｜parameter> and </｜DSML｜inv> instead of </｜DSML｜invoke>, leaving tool_calls unparsed. Pass comb.stride(0), comb.stride(2), comb.stride(1) so out_h strides the OUT axis (stride(2)) and src_h the SRC axis (stride(1)), yielding matmul(comb.T, residual). Verified end-to-end: tool_choice='auto' now returns parsed tool_calls with correct closing tags. The test reference _reference_post_expand encoded the same transposed mix (matmul(comb, residual)); corrected to matmul(comb.T, residual). Co-Authored-By: Claude <noreply@anthropic.com>

grimoire · 2026-08-24T15:50:03Z

{
  "model": "/mnt/shared-storage-gpfs2/gpfs2-shared-public/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731/snapshots/7872f01b1d1fe23eabc4c98b48bffcef5a386062/",
  "messages": [
    {
      "role": "user",
      "content": "What is the weather like in San Francisco, CA? Use the weather tool."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string",
              "description": "The city to find the weather for, e.g. San Francisco"
            },
            "state": {
              "type": "string",
              "description": "The state abbreviation, e.g. CA"
            },
            "unit": {
              "type": "string",
              "description": "The unit for temperature",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["city", "state"]
        }
      }
    }
  ],
  "tool_choice": "none",
  "temperature": 0,
  "max_completion_tokens": 1024
}

The response log shows:

2026-08-24 04:40:42,982 - lmdeploy - REQUEST - logger.py:55 - session=0, response='\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke>\n<｜DSML｜invoke name="get_current_weather">\n<｜DSML｜parameter name="city" string="true">San Francisco</｜DSML｜parameter>\n<｜DSML｜parameter parameter="state"="CA">CA</｜DSML｜parameter>\n<parameter name="unit">fahrenheit</parameter>\n</｜DSML｜parameter>\n</｜DSML｜inv>\n</｜DSML｜invoke>\n</｜DSML｜tool>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>'

This does not appear to be a valid tool_call response. Could you please help investigate whether this is caused by the model, the chat template, the inference pipeline, or something else?

fixed

lvhan028 · 2026-08-26T04:05:12Z

The root cause of test_docker_image failure is brought by #4853

#4853 added flashinfer-python==0.6.15.post1 to requirements/runtime_cuda.txt on main.

FlashInfer pulls in nvidia-cutlass-dsl, which installs nvidia-cutlass-dsl-libs-cu12.
docker/install.sh has a cu130 validation step that rejects any package ending in -cu12

cc @CUHKSZzxy

grimoire added 2 commits August 18, 2026 16:48

fix: update DeepSeek V4 for native Transformers config

c518db4

refactor: own DSV4 warmup shape in indexer backend

e939d5e

Copilot AI lite review requested due to automatic review settings August 18, 2026 09:21

Copilot started reviewing on behalf of grimoire August 18, 2026 09:21 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

lvhan028 added the Bug:P1 label Aug 20, 2026

lvhan028 self-requested a review August 20, 2026 02:57

grimoire added 2 commits August 20, 2026 21:14

fix: restore HF config fallback registration

1643962

fix: synchronize guided token acceptance

dd4d430

install tile-kernels in docker/install.sh

fe13b4b

grimoire and others added 3 commits August 24, 2026 14:32

install tile-kernels in docker/install.sh

ffc374c

Co-Authored-By: Claude <noreply@anthropic.com>

install tile-kernels only under the cu13 branch

0d431fc

tile-kernels depends on a cu13 tilelang build, so gate the pip install on CUDA_VERSION_SHORT == cu13* instead of installing it unconditionally. Co-Authored-By: Claude <noreply@anthropic.com>

fix

da68946

lvhan028 approved these changes Aug 26, 2026

View reviewed changes

lvhan028 merged commit c4fdac4 into InternLM:main Aug 26, 2026
6 of 14 checks passed

Navigation Menu

Uh oh!

Fix/dsv4 native transformers warmup - #4878

Fix/dsv4 native transformers warmup#4878
lvhan028 merged 9 commits into
InternLM:mainfrom
grimoire:fix/dsv4-native-transformers-warmup

grimoire commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

lvhan028 commented Aug 24, 2026

Uh oh!

lvhan028 commented Aug 24, 2026 •

edited

Loading

Uh oh!

grimoire commented Aug 24, 2026

Uh oh!

lvhan028 commented Aug 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

grimoire commented Aug 18, 2026

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

lvhan028 commented Aug 24, 2026

Uh oh!

lvhan028 commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grimoire commented Aug 24, 2026

Uh oh!

lvhan028 commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lvhan028 commented Aug 24, 2026 •

edited

Loading

lvhan028 commented Aug 26, 2026 •

edited

Loading