# Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

DevFeed: [Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel](<https://devfeed.tech/articles/accelerating-transformers-fine-tuning-with-nvidia-nemo-automodel-7374.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel>)

Author: Adil Asif; Alexandros Koumparoulis; Wenwen Gao; Sylendran Arunagiri; David Messina; Bernard Nguyen

Published: 2026-06-24T16:00:13Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [NeMo](<https://devfeed.tech/topics/nemo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [sglang](<https://devfeed.tech/topics/sglang.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [building](<https://devfeed.tech/tags/building.md>), [compute](<https://devfeed.tech/tags/compute.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nemo](<https://devfeed.tech/tags/nemo.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open](<https://devfeed.tech/tags/open.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

This article explains how NVIDIA NeMo AutoModel accelerates fine-tuning of mixture-of-experts models by extending Transformers v5 with Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels. It describes API compatibility, distributed execution, dynamic weight loading, and reported gains of 3.4-3.7x higher training throughput and 29-32% lower GPU memory use.

## Source excerpt

NVIDIA NeMo AutoModel is an open library part of the NVIDIA NeMo framework for building custom generative AI models at scale. NeMo AutoModel builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5's dynamic weight loading to bring those optimizations to a broad and growing set of model families.