Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ specific language governing permissions and limitations under the License. -->

# AutoencoderKLHunyuanVideo15

The 3D variational autoencoder (VAE) model with KL loss used in [HunyuanVideo1.5](https://github.com/Tencent/HunyuanVideo1-1.5) by Tencent.
The 3D variational autoencoder (VAE) model with KL loss used in [HunyuanVideo1.5](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5) by Tencent.

The model can be loaded with the following code snippet.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/models/autoencoder_same.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ specific language governing permissions and limitations under the License.

# AutoencoderSAME

The **SAME** (Semantically-Aligned Music Encoder) autoencoder is used by [Stable Audio 3](https://stability.ai/news/stable-audio-3)
The **SAME** (Semantically-Aligned Music Encoder) autoencoder is used by [Stable Audio 3](https://stability.ai/news-updates/meet-stable-audio-3-the-model-family-built-for-artistic-experimentation-with-open-weight-models)
to compress stereo audio waveforms into a compact latent sequence and reconstruct them.

The encoder stacks [`SAMETransformerResamplingBlock`] modules, each of which groups a fixed number of audio
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/models/autoencoderkl_cogvideox.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ specific language governing permissions and limitations under the License. -->

# AutoencoderKLCogVideoX

The 3D variational autoencoder (VAE) model with KL loss used in [CogVideoX](https://github.com/THUDM/CogVideo) was introduced in [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://github.com/THUDM/CogVideo/blob/main/resources/CogVideoX.pdf) by Tsinghua University & ZhipuAI.
The 3D variational autoencoder (VAE) model with KL loss used in [CogVideoX](https://github.com/THUDM/CogVideo) was introduced in [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://huggingface.co/papers/2408.06072) by Tsinghua University & ZhipuAI.

The model can be loaded with the following code snippet.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/models/cogvideox_transformer3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ specific language governing permissions and limitations under the License. -->

# CogVideoXTransformer3DModel

A Diffusion Transformer model for 3D data from [CogVideoX](https://github.com/THUDM/CogVideo) was introduced in [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://github.com/THUDM/CogVideo/blob/main/resources/CogVideoX.pdf) by Tsinghua University & ZhipuAI.
A Diffusion Transformer model for 3D data from [CogVideoX](https://github.com/THUDM/CogVideo) was introduced in [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://huggingface.co/papers/2408.06072) by Tsinghua University & ZhipuAI.

The model can be loaded with the following code snippet.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ specific language governing permissions and limitations under the License. -->

# HunyuanVideo15Transformer3DModel

A Diffusion Transformer model for 3D video-like data used in [HunyuanVideo1.5](https://github.com/Tencent/HunyuanVideo1-1.5).
A Diffusion Transformer model for 3D video-like data used in [HunyuanVideo1.5](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5).

The model can be loaded with the following code snippet.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/models/stable_audio_3_transformer.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ specific language governing permissions and limitations under the License.
# StableAudio3DiTModel

A rectified-flow velocity-prediction Diffusion Transformer (DiT) for audio generation, used in
[Stable Audio 3](https://stability.ai/news/stable-audio-3).
[Stable Audio 3](https://stability.ai/news-updates/meet-stable-audio-3-the-model-family-built-for-artistic-experimentation-with-open-weight-models).

Each [`StableAudio3DiTBlock`] performs:

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/pipelines/lumina.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ specific language governing permissions and limitations under the License.
# Lumina-T2X
![concepts](https://github.com/Alpha-VLLM/Lumina-T2X/assets/54879512/9f52eabb-07dc-4881-8257-6d8a5f2a0a5a)

[Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT](https://github.com/Alpha-VLLM/Lumina-T2X/blob/main/assets/lumina-next.pdf) from Alpha-VLLM, OpenGVLab, Shanghai AI Laboratory.
[Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT](https://huggingface.co/papers/2406.18583) from Alpha-VLLM, OpenGVLab, Shanghai AI Laboratory.

The abstract from the paper is:

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/api/pipelines/pix2pix.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The abstract from the paper is:

*We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.*

You can find additional information about InstructPix2Pix on the [project page](https://www.timothybrooks.com/instruct-pix2pix), [original codebase](https://github.com/timothybrooks/instruct-pix2pix), and try it out in a [demo](https://huggingface.co/spaces/timbrooks/instruct-pix2pix).
You can find additional information about InstructPix2Pix in the [paper](https://huggingface.co/papers/2211.09800), [original codebase](https://github.com/timothybrooks/instruct-pix2pix), and try it out in a [demo](https://huggingface.co/spaces/timbrooks/instruct-pix2pix).

> [!TIP]
> Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reusing-models-in-multiple-pipelines) section to learn how to efficiently load the same components into multiple pipelines.
Expand Down
8 changes: 4 additions & 4 deletions docs/source/en/optimization/fp16.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,10 +252,10 @@ The [diffusers-torchao](https://github.com/sayakpaul/diffusers-torchao#benchmark

[Kernels](https://huggingface.co/docs/kernels/index) is a library for building, distributing, and loading optimized compute kernels on the [Hub](https://huggingface.co/kernels-community). It supports [attention](./attention_backends#setattentionbackend) kernels and custom CUDA kernels for operations like RMSNorm, GEGLU, RoPE, and AdaLN.

The [Diffusers Pipeline Integration](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/references/diffusers-integration.md) guide shows how to integrate a kernel with the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/SKILL.md) skill. This skill enables an agent, like Claude or Codex, to write custom kernels targeted towards a specific model and your hardware.
The [Diffusers Pipeline Integration](https://github.com/huggingface/kernels/blob/main/kernel-builder/skills/cuda-kernels/references/diffusers-integration.md) guide shows how to integrate a kernel with the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/kernel-builder/skills/cuda-kernels/SKILL.md) skill. This skill enables an agent, like Claude or Codex, to write custom kernels targeted towards a specific model and your hardware.

> [!TIP]
> Install the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/SKILL.md) skill to teach an agent how to write a kernel. The [Custom kernels for all from Codex and Claude](https://huggingface.co/blog/custom-cuda-kernels-agent-skills) blog post covers this in more detail.
> Install the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/kernel-builder/skills/cuda-kernels/SKILL.md) skill to teach an agent how to write a kernel. The [Custom kernels for all from Codex and Claude](https://huggingface.co/blog/custom-cuda-kernels-agent-skills) blog post covers this in more detail.

For example, a custom RMSNorm kernel (generated by the `add cuda-kernels` skill) with [torch.compile](#torchcompile) speeds up LTX-Video generation 1.43x on an H100.

Expand All @@ -268,9 +268,9 @@ For example, a custom RMSNorm kernel (generated by the `add cuda-kernels` skill)

## Dynamic quantization

[Dynamic quantization](https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html) improves inference speed by reducing precision to enable faster math operations. This particular type of quantization determines how to scale the activations based on the data at runtime rather than using a fixed scaling factor. As a result, the scaling factor is more accurately aligned with the data.
[Dynamic quantization](https://docs.pytorch.org/ao/main/api_reference/generated/torchao.quantization.Int8DynamicActivationInt8WeightConfig.html) improves inference speed by reducing precision to enable faster math operations. This particular type of quantization determines how to scale the activations based on the data at runtime rather than using a fixed scaling factor. As a result, the scaling factor is more accurately aligned with the data.

The example below applies [dynamic int8 quantization](https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html) to the UNet and VAE with the [torchao](../quantization/torchao) library.
The example below applies [dynamic int8 quantization](https://docs.pytorch.org/ao/main/api_reference/generated/torchao.quantization.Int8DynamicActivationInt8WeightConfig.html) to the UNet and VAE with the [torchao](../quantization/torchao) library.

> [!TIP]
> Refer to our [torchao](../quantization/torchao) docs to learn more about how to use the Diffusers torchao integration.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/optimization/pruna.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ Now that you have seen how to optimize and evaluate your models, you can start u

## Reference

- [Pruna](https://github.com/pruna-ai/pruna)
- [Pruna](https://github.com/PrunaAI/pruna)
- [Pruna optimization](https://docs.pruna.ai/en/stable/docs_pruna/user_manual/configure.html#configure-algorithms)
- [Pruna evaluation](https://docs.pruna.ai/en/stable/docs_pruna/user_manual/evaluate.html)
- [Pruna tutorials](https://docs.pruna.ai/en/stable/docs_pruna/tutorials/index.html)
Expand Down
4 changes: 2 additions & 2 deletions docs/source/en/quantization/torchao.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ Make sure Pytorch 2.5+ and torchao are installed with the command below.
uv pip install -U torch torchao
```

Each quantization dtype is available as a separate instance of a [AOBaseConfig](https://docs.pytorch.org/ao/main/api_ref_quantization.html#inference-apis-for-quantize) class. This provides more flexible configuration options by exposing more available arguments.
Each quantization dtype is available as a separate instance of a [AOBaseConfig](https://docs.pytorch.org/ao/main/api_reference/generated/torchao.core.config.AOBaseConfig.html) class. This provides more flexible configuration options by exposing more available arguments.

Pass the `AOBaseConfig` of a quantization dtype, like [Int4WeightOnlyConfig](https://docs.pytorch.org/ao/main/generated/torchao.quantization.Int4WeightOnlyConfig) to [`TorchAoConfig`] in [`~ModelMixin.from_pretrained`].
Pass the `AOBaseConfig` of a quantization dtype, like [Int4WeightOnlyConfig](https://docs.pytorch.org/ao/main/api_reference/generated/torchao.quantization.Int4WeightOnlyConfig.html) to [`TorchAoConfig`] in [`~ModelMixin.from_pretrained`].

```py
import torch
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/training/controlnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ As with the script parameters, a general walkthrough of the training script is p
The training script has a [`make_train_dataset`](https://github.com/huggingface/diffusers/blob/64603389da01082055a901f2883c4810d1144edb/examples/controlnet/train_controlnet.py#L582) function for preprocessing the dataset with image transforms and caption tokenization. You'll see that in addition to the usual caption tokenization and image transforms, the script also includes transforms for the conditioning image.

> [!TIP]
> If you're streaming a dataset on a TPU, performance may be bottlenecked by the 🤗 Datasets library which is not optimized for images. To ensure maximum throughput, you're encouraged to explore other dataset formats like [WebDataset](https://webdataset.github.io/webdataset/), [TorchData](https://github.com/pytorch/data), and [TensorFlow Datasets](https://www.tensorflow.org/datasets/tfless_tfds).
> If you're streaming a dataset on a TPU, performance may be bottlenecked by the 🤗 Datasets library which is not optimized for images. To ensure maximum throughput, you're encouraged to explore other dataset formats like [WebDataset](https://github.com/webdataset/webdataset), [TorchData](https://github.com/pytorch/data), and [TensorFlow Datasets](https://www.tensorflow.org/datasets/tfless_tfds).

```py
conditioning_image_transforms = transforms.Compose(
Expand Down
6 changes: 3 additions & 3 deletions docs/source/en/training/nemo_automodel.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Preprocessing produces a cache directory organized by resolution bucket. NeMo Au
```

> [!TIP]
> See the [Diffusion Dataset Preparation](https://docs.nvidia.com/nemo/automodel/latest/guides/diffusion/dataset.html) guide for caption formats, input data requirements, and all available preprocessing arguments.
> See the [Diffusion Dataset Preparation](https://docs.nvidia.com/nemo/automodel/latest/datasets/diffusion-dataset) guide for caption formats, input data requirements, and all available preprocessing arguments.

## Training configuration

Expand Down Expand Up @@ -372,7 +372,7 @@ This integration provides several benefits for Diffusers users:
## Resources

- [NeMo Automodel GitHub](https://github.com/NVIDIA-NeMo/Automodel)
- [Diffusion Fine-Tuning Guide](https://docs.nvidia.com/nemo/automodel/latest/guides/diffusion/finetune.html)
- [Diffusion Dataset Preparation](https://docs.nvidia.com/nemo/automodel/latest/guides/diffusion/dataset.html)
- [Diffusion Fine-Tuning Guide](https://docs.nvidia.com/nemo/automodel/latest/recipes-e2e-examples/diffusion-fine-tuning)
- [Diffusion Dataset Preparation](https://docs.nvidia.com/nemo/automodel/latest/datasets/diffusion-dataset)
- [Diffusion Model Coverage](https://docs.nvidia.com/nemo/automodel/latest/model-coverage/diffusion.html)
- [NeMo Automodel for Transformers (LLM/VLM fine-tuning)](https://huggingface.co/docs/transformers/en/community_integrations/nemo_automodel_finetuning)
2 changes: 1 addition & 1 deletion docs/source/en/training/t2i_adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ If you want to learn more about how the training loop works, check out the [Unde

Now you’re ready to launch the training script! 🚀

For this example training, you'll use the [fusing/fill50k](https://huggingface.co/datasets/fusing/fill50k) dataset. You can also create and use your own dataset if you want (see the [Create a dataset for training](https://moon-ci-docs.huggingface.co/docs/diffusers/pr_5512/en/training/create_dataset) guide).
For this example training, you'll use the [fusing/fill50k](https://huggingface.co/datasets/fusing/fill50k) dataset. You can also create and use your own dataset if you want (see the [Create a dataset for training](create_dataset) guide).

Set the environment variable `MODEL_DIR` to a model id on the Hub or a path to a local model and `OUTPUT_DIR` to where you want to save the model.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/using-diffusers/text-img2vid.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ export_to_video(output, "output.mp4", fps=16)

## Inference speed

[torch.compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial_.html) can speedup inference by using optimized kernels. Compilation takes longer the first time, but once compiled, it is much faster. It is best to compile the pipeline once, and then use the pipeline multiple times without changing anything. A change, such as in the image size, triggers recompilation.
[torch.compile](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) can speedup inference by using optimized kernels. Compilation takes longer the first time, but once compiled, it is much faster. It is best to compile the pipeline once, and then use the pipeline multiple times without changing anything. A change, such as in the image size, triggers recompilation.

The example below compiles the transformer in the pipeline and uses the `"max-autotune"` mode to maximize performance.

Expand Down
Loading