# Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication

DevFeed: [Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication](<https://devfeed.tech/articles/zero-sm-nccl-a-systematic-guide-to-designing-and-deploying-sm-free-gpu-communication-134894.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18075>)

Author: Zhenhao He, Siyuan Shen, Ke Wen, Prajwal Singhania, Shiqing Fan, Youngeun Kwon, Sangkug Lym, Vasudevan Rengasamy, Vishnuswaroop Ramesh, Yicheng Qiang, Craig Belusar, Jeff Hammond, Torsten Hoefler

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [NCCL](<https://devfeed.tech/topics/nccl.md>), [rdma](<https://devfeed.tech/topics/rdma.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [apis](<https://devfeed.tech/tags/apis.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [building](<https://devfeed.tech/tags/building.md>), [communication](<https://devfeed.tech/tags/communication.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [deep](<https://devfeed.tech/tags/deep.md>), [designing](<https://devfeed.tech/tags/designing.md>), [domain](<https://devfeed.tech/tags/domain.md>), [domains](<https://devfeed.tech/tags/domains.md>), [free](<https://devfeed.tech/tags/free.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [guide](<https://devfeed.tech/tags/guide.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [sm](<https://devfeed.tech/tags/sm.md>), [zero](<https://devfeed.tech/tags/zero.md>)

## AI overview

Zero-SM NCCL avoids SM kernels by synchronizing with CUDA stream memory operations, using copy engines for transfers within a scale-up domain, and driving network RDMA through a stream-ordered CPU proxy. It supports AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8, collectives achieved up to 1.36x NCCL's legacy AllGather bandwidth and reduced median overlapped GEMM slowdown from 33% to 4%, with higher per-operation latency. In Megatron-LM, FSDP improved by up to 3.5% and pipeline parallelism by up to 15% with one-sided puts. The gains depend on workload: the approach benefits large transfers in GPU-bound workloads and deep pipelines with large bubbles, but loses on small messages and host-bound workloads.

## Source excerpt

arXiv:2611.18075v1 Announce Type: new Abstract: GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.