Add Kandinsky S2V 1.0 (speech-to-video) as a modular pipeline

docs/source/en/_toctree.yml

@@ -381,6 +381,8 @@
         title: JoyImageEditTransformer3DModel
       - local: api/models/kandinsky6_transformer
         title: Kandinsky 6 Transformers
+      - local: api/models/kandinsky_s2v_transformer
+        title: KandinskyS2VTransformer3DModel
       - local: api/models/krea2_transformer2d
         title: Krea2Transformer2DModel
       - local: api/models/latte_transformer3d
@@ -757,6 +759,8 @@
         title: Kandinsky 5.0 Video
       - local: api/pipelines/kandinsky6
         title: Kandinsky 6
+      - local: api/pipelines/kandinsky_s2v
+        title: Kandinsky S2V 1.0
       - local: api/pipelines/latte
         title: Latte
       - local: api/pipelines/ltx2

docs/source/en/api/models/kandinsky_s2v_transformer.md

@@ -0,0 +1,22 @@
+<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# KandinskyS2VTransformer3DModel
+
+The Diffusion Transformer (DiT) of Kandinsky S2V 1.0, conditioned on a reference
+frame and a speech track.
+
+## KandinskyS2VTransformer3DModel
+
+[[autodoc]] KandinskyS2VTransformer3DModel
+  - all
+  - forward

docs/source/en/api/pipelines/kandinsky_s2v.md

@@ -0,0 +1,137 @@
+<!--Copyright 2026 The Kandinsky Team and The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Kandinsky S2V 1.0
+
+Kandinsky S2V 1.0 generates an avatar video from a reference frame and a speech
+track. The person in the frame is animated to match the speech — the whole figure,
+not only the head. The speech is a condition, not an output: the model does not
+generate audio, and the soundtrack of the result is the track you pass in.
+
+Both conditions go through [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B):
+its hidden states encode the prompt and its audio tower encodes the speech, which
+enters the transformer through cross-attention adapters aligned frame by frame, so
+each latent frame attends only to the speech it covers. A clip is generated in
+chunks of 5 s (125 frames at 25 fps), each carrying FramePack memory of the
+previous ones, so longer tracks stay consistent across chunk boundaries.
+
+Kandinsky S2V 1.0 is integrated as a [modular pipeline](../../modular_diffusers/overview)
+only; there is no standard `DiffusionPipeline` class for it.
+
+## Checkpoints
+
+| Checkpoint | Transformer | Blocks | Scheduler | Steps | Guidance |
+|---|---|---|---|---|---|
+| [Lite][lite] | 2B | [`KandinskyS2VAutoBlocks`] | [`FlowMatchEulerDiscreteScheduler`] | 50 | 3.0 |
+| [Pro][pro] | 21B | [`KandinskyS2VAutoBlocks`] | [`FlowMatchEulerDiscreteScheduler`] | 50 | 3.0 |
+| [Lite distilled][lite-distill] | 2B | [`KandinskyS2VDistilledAutoBlocks`] | [`PiflowScheduler`] | 16 | off |
+| [Pro distilled][pro-distill] | 21B | [`KandinskyS2VDistilledAutoBlocks`] | [`PiflowScheduler`] | 16 | off |
+
+The distilled checkpoints configure `PiflowScheduler` with `shift=10.0` and `n_grid=5`, not its defaults.
+
+[lite]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Lite-Diffusers
+[pro]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers
+[lite-distill]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers
+[pro-distill]: https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro-distill-Diffusers
+
+`ModularPipeline.from_pretrained` picks the blocks from the checkpoint. The steps
+default to the values above. Classifier-free guidance is the `guider` component,
+not a call argument: the base checkpoints ship it at 3.0. The distilled ones have
+no guider and no `negative_prompt`, because guidance is already part of the student.
+
+## Usage
+
+```py
+import librosa
+import torch
+from diffusers import ModularPipeline
+from diffusers.utils import encode_video, load_image
+
+pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers")
+pipe.load_components(dtype=torch.bfloat16)
+pipe.to("cuda")
+
+image = load_image("reference.png")
+audio, _ = librosa.load("speech.wav", sr=16000)  # a mono waveform at 16 kHz, for the model
+speech, sample_rate = librosa.load("speech.wav", sr=None)  # the original track, added to the output video
+
+video = pipe(image=image, audio=audio, output="videos")[0]
+encode_video(
+    video,
+    fps=25,
+    output_path="avatar.mp4",
+    audio=torch.from_numpy(speech)[None],
+    audio_sample_rate=sample_rate,
+)
+```
+
+The Pro checkpoints do not fit an 80 GB GPU next to the 7B speech and text encoder
+when every component is on the device. Offload them through a `ComponentsManager`:
+
+```py
+import librosa
+import torch
+from diffusers import ComponentsManager, ModularPipeline
+from diffusers.utils import encode_video, load_image
+
+manager = ComponentsManager()
+pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers", components_manager=manager)
+pipe.load_components(dtype=torch.bfloat16)
+manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB")
+
+image = load_image("reference.png")
+audio, _ = librosa.load("speech.wav", sr=16000)
+speech, sample_rate = librosa.load("speech.wav", sr=None)
+
+video = pipe(image=image, audio=audio, output="videos")[0]
+encode_video(
+    video,
+    fps=25,
+    output_path="avatar.mp4",
+    audio=torch.from_numpy(speech)[None],
+    audio_sample_rate=sample_rate,
+)
+```
+
+The margin is the memory kept free beyond the weights when a model is moved to the GPU.
+It has to hold the VAE decode of a chunk, about 33 GiB at 512x768: with the default 3 GB
+the transformer stays on the GPU through the decode, which then runs out of memory. With
+40 GiB the peak is about 50 GiB on an 80 GB GPU. Enable the offload after `load_components`,
+so it applies to every component.
+
+## Notes
+
+- `image` is a PIL image and `audio` a mono float waveform in [-1, 1] at 16 kHz, as a tensor
+  or an array shaped `(samples,)`; the pipeline does not read files.
+- The top-level blocks (`text_encoder`, `vae_encoder`, `denoise`, `decode`) run on their own too;
+  pass the same `num_latent_frames` and `fps` to each when you run them separately.
+- The number of chunks follows the length of the speech track unless `num_chunks`
+  is passed. The video is trimmed to the track either way, so chunks past its end
+  are computed and dropped, with a warning.
+- With guidance, the unconditional pass swaps in the negative prompt *and* silences
+  the speech, so the guidance is joint over text and speech.
+
+## KandinskyS2VModularPipeline
+
+[[autodoc]] KandinskyS2VModularPipeline
+
+## KandinskyS2VAutoBlocks
+
+[[autodoc]] KandinskyS2VAutoBlocks
+
+## KandinskyS2VDistilledModularPipeline
+
+[[autodoc]] KandinskyS2VDistilledModularPipeline
+
+## KandinskyS2VDistilledAutoBlocks
+
+[[autodoc]] KandinskyS2VDistilledAutoBlocks

docs/source/en/api/schedulers/piflow.md

@@ -13,7 +13,8 @@ specific language governing permissions and limitations under the License.
 # PiflowScheduler
 
 `PiflowScheduler` is the few-step scheduler of the distilled Kandinsky 6 checkpoints, both the
-text/image-to-video-and-audio model and the video super-resolution model. It implements
+text/image-to-video-and-audio model and the video super-resolution model, and of the distilled Kandinsky S2V 1.0
+checkpoints. It implements
 [π-Flow](https://huggingface.co/papers/2510.14974): the transformer predicts `n_grid` denoised estimates per latent
 channel at a small number of grid points, and the scheduler integrates a network-free policy between them.
 

src/diffusers/__init__.py

@@ -329,6 +329,7 @@
             "Kandinsky6SRTransformer3DModel",
             "Kandinsky6SRVAE",
             "Kandinsky6Transformer3DModel",
+            "KandinskyS2VTransformer3DModel",
             "Krea2Transformer2DModel",
             "LatteTransformer3DModel",
             "LongCatAudioDiTTransformer",
@@ -586,6 +587,10 @@
             "HunyuanVideo15ModularPipeline",
             "Ideogram4AutoBlocks",
             "Ideogram4ModularPipeline",
+            "KandinskyS2VAutoBlocks",
+            "KandinskyS2VDistilledAutoBlocks",
+            "KandinskyS2VDistilledModularPipeline",
+            "KandinskyS2VModularPipeline",
             "Krea2AutoBlocks",
             "Krea2ModularPipeline",
             "Krea2TurboAutoBlocks",
@@ -1256,6 +1261,7 @@
             Kandinsky6SRTransformer3DModel,
             Kandinsky6SRVAE,
             Kandinsky6Transformer3DModel,
+            KandinskyS2VTransformer3DModel,
             Krea2Transformer2DModel,
             LatteTransformer3DModel,
             LongCatAudioDiTTransformer,
@@ -1491,6 +1497,10 @@
             HunyuanVideo15ModularPipeline,
             Ideogram4AutoBlocks,
             Ideogram4ModularPipeline,
+            KandinskyS2VAutoBlocks,
+            KandinskyS2VDistilledAutoBlocks,
+            KandinskyS2VDistilledModularPipeline,
+            KandinskyS2VModularPipeline,
             Krea2AutoBlocks,
             Krea2ModularPipeline,
             Krea2TurboAutoBlocks,

src/diffusers/models/__init__.py

@@ -149,6 +149,7 @@
     _import_structure["transformers.transformer_kandinsky"] = ["Kandinsky5Transformer3DModel"]
     _import_structure["transformers.transformer_kandinsky6"] = ["Kandinsky6Transformer3DModel"]
     _import_structure["transformers.transformer_kandinsky6_sr"] = ["Kandinsky6SRTransformer3DModel"]
+    _import_structure["transformers.transformer_kandinsky_s2v"] = ["KandinskyS2VTransformer3DModel"]
     _import_structure["transformers.transformer_krea2"] = ["Krea2Transformer2DModel"]
     _import_structure["transformers.transformer_longcat_audio_dit"] = ["LongCatAudioDiTTransformer"]
     _import_structure["transformers.transformer_longcat_image"] = ["LongCatImageTransformer2DModel"]
@@ -313,6 +314,7 @@
             Kandinsky5Transformer3DModel,
             Kandinsky6SRTransformer3DModel,
             Kandinsky6Transformer3DModel,
+            KandinskyS2VTransformer3DModel,
             Krea2Transformer2DModel,
             LatteTransformer3DModel,
             LongCatAudioDiTTransformer,

src/diffusers/models/transformers/__init__.py

@@ -49,6 +49,7 @@
     from .transformer_kandinsky import Kandinsky5Transformer3DModel
     from .transformer_kandinsky6 import Kandinsky6Transformer3DModel
     from .transformer_kandinsky6_sr import Kandinsky6SRTransformer3DModel
+    from .transformer_kandinsky_s2v import KandinskyS2VTransformer3DModel
     from .transformer_krea2 import Krea2Transformer2DModel
     from .transformer_longcat_audio_dit import LongCatAudioDiTTransformer
     from .transformer_longcat_image import LongCatImageTransformer2DModel