# PyTorch 2.15 tackles the 'one bad GPU' problem

DevFeed: [PyTorch 2.15 tackles the 'one bad GPU' problem](<https://devfeed.tech/articles/pytorch-2-15-tackles-the-one-bad-gpu-problem-122495.md>)

Original publisher: [Read original article](<https://www.devclass.com/devops/2026/11/07/pytorch-215-tackles-the-one-bad-gpu-problem/5308234>)

Author: Joab Jackson

Published: 2026-11-07T18:38:00Z

Content type: news

Language: en

Sources: [DevClass.com](<https://devfeed.tech/sources/devclass-com.md>)

Topics: [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [api](<https://devfeed.tech/tags/api.md>), [apple](<https://devfeed.tech/tags/apple.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [communication](<https://devfeed.tech/tags/communication.md>), [communications](<https://devfeed.tech/tags/communications.md>), [computing](<https://devfeed.tech/tags/computing.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [deadlocks](<https://devfeed.tech/tags/deadlocks.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [job](<https://devfeed.tech/tags/job.md>), [ml](<https://devfeed.tech/tags/ml.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

PyTorch 2.15 rewrites its NCCL backend to improve diagnosis and recovery in multi-GPU jobs. The former ProcessGroupNCCL could create many communicators per process group, contributing to intermittent deadlocks and unpredictable performance; the new backend limits each group to one communicator at initialization, so programs needing parallelism must dispatch more groups. It also supports easier reconfiguration when a GPU is slow or fails: a third-party orchestrator can exclude it and assign a replacement worker. A new flight recorder stores operation snapshots for inspection after failures. The legacy backend remains available as a rollback in 2.15 and is slated for removal in 2.16 if the replacement works as expected. Other reported additions include an asynchronous one-way Python transport API and parallel-processing improvements for Apple hardware and Nvidia Blackwell GPUs.

## Source excerpt

ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job ...