# The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

DevFeed: [The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch](<https://devfeed.tech/articles/the-endgame-of-fsdp-communication-layout-aware-collectives-on-copy-engines-rdma-and-nvswitch-107569.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38228>)

Author: Zejian Wang, Yifan Zhao, Kaihui Ren, Xiaoling Zhou, Yubin Wang, Chenhao Guan, Lin Zhang, Yufeng Li, Zhikang Mo, Zhuoru Ma, Yingtong Xiong, Jihu Guo, Yiqiao Liao, Chengru Song

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [rdma](<https://devfeed.tech/topics/rdma.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>)

Tags: [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [engines](<https://devfeed.tech/tags/engines.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [layout](<https://devfeed.tech/tags/layout.md>), [layouts](<https://devfeed.tech/tags/layouts.md>), [llama](<https://devfeed.tech/tags/llama.md>), [magnitude](<https://devfeed.tech/tags/magnitude.md>), [memory](<https://devfeed.tech/tags/memory.md>), [multicast](<https://devfeed.tech/tags/multicast.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [nvswitch](<https://devfeed.tech/tags/nvswitch.md>), [rdma](<https://devfeed.tech/tags/rdma.md>)

## AI overview

TIDE targets two FSDP2 costs: copying between per-parameter tensors and flat collective buffers, and communication kernels competing with compute. It folds placement into collective addressing while preserving FSDP2 storage and compute kernels. TIDE-AG writes shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA without using SMs for transport or placement. TIDE-RS reads per-parameter bf16 gradients and produces fp32 shards through copy engines, short-lived CTAs for node-local accumulation, and NVSwitch for cross-node reduction; its two-phase pipeline separates gradient reads from network progress, with optional Slot-Reuse. In reported 64-GPU experiments across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B, TIDE cut step time by up to 27.1% versus Stock FSDP2 and 6.0% versus FSDP2+SymmMem. The abstract also reports 12.5-14.5x lower reduce-scatter SM time and up to 4.7 GiB less peak memory than FSDP2+SymmMem. These are reported maxima; the supplied evidence is an abstract, not full paper text.

## Source excerpt

arXiv:2610.38228v1 Announce Type: new Abstract: Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's.