# From DeepSpeed to FSDP and Back Again with Hugging Face Accelerate

DevFeed: [From DeepSpeed to FSDP and Back Again with Hugging Face Accelerate](<https://devfeed.tech/articles/from-deepspeed-to-fsdp-and-back-again-with-hugging-face-accelerate-7163.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/deepspeed-to-fsdp-and-back>)

Author: Yu Chin Fabian Lim; aldo pareja; Zachary Mueller; Stas Bekman

Published: 2024-06-13T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [research](<https://devfeed.tech/tags/research.md>), [torch](<https://devfeed.tech/tags/torch.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

The article compares DeepSpeed and PyTorch FSDP mixed-precision training behavior. It attributes initially different convergence results to DeepSpeed's internal upcasting of master weights to full precision, then discusses the memory and optimizer implications of this design versus FSDP.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.