# Auto repair for Together GPU Clusters: from GPU fault to healthy node

DevFeed: [Auto repair for Together GPU Clusters: from GPU fault to healthy node](<https://devfeed.tech/articles/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-103670.md>)

Original publisher: [Read original article](<https://www.together.ai/blog/auto-repair-for-together-gpu-clusters>)

Author: Pavneet Ahluwalia, Dongsheng Zhang, Barbara Kazanowska

Published: 2026-10-27T00:00:00Z

Content type: article

Language: en

Sources: [Together.ai](<https://devfeed.tech/sources/together-ai.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [account](<https://devfeed.tech/tags/account.md>), [actions](<https://devfeed.tech/tags/actions.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [also](<https://devfeed.tech/tags/also.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [code](<https://devfeed.tech/tags/code.md>), [fabric](<https://devfeed.tech/tags/fabric.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-clusters](<https://devfeed.tech/tags/gpu-clusters.md>), [node](<https://devfeed.tech/tags/node.md>), [repair](<https://devfeed.tech/tags/repair.md>), [run](<https://devfeed.tech/tags/run.md>), [works](<https://devfeed.tech/tags/works.md>)

## AI overview

Together GPU Clusters' auto-repair pipeline uses failed health checks and supporting evidence to recommend the lightest mapped fix; `NotReady` alone never triggers repair. Detection runs inside VMs and on hosts, while policy controls which faults run automatically and how long repairs wait for workloads to become idle. Repairs proceed through cordon, wait, drain, action and rejoin, with escalation after repeated failures, evidence and capacity gates for migrations, and safeguards against repair storms. Customers can submit repairs through the API or console; node-label remediation contracts are in preview. After migration or removal, the vacated machine is automatically tested and returned to service, ticketed after explicit failure, or restored to its prior state if validation is aborted. Reboots preserve local scratch data; quick reprovisioning loses it. The excerpt also describes host power cycling as a preview feature.

## Source excerpt

Together GPU Clusters auto repair detects GPU, host and fabric faults, picks the lightest fix, and lets you decide per fault and per Xid code which repairs run automatically. Here is how it works.