# Diagnose Distributed AI Workload Failures with NVIDIA Attribution

DevFeed: [Diagnose Distributed AI Workload Failures with NVIDIA Attribution](<https://devfeed.tech/articles/diagnose-distributed-ai-workload-failures-with-nvidia-attribution-113259.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/>)

Author: Elizabeth Goodman

Published: 2026-11-02T23:39:34Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>)

Tags: [accelerated-computing-libraries](<https://devfeed.tech/tags/accelerated-computing-libraries.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [diagnose](<https://devfeed.tech/tags/diagnose.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [error](<https://devfeed.tech/tags/error.md>), [errors](<https://devfeed.tech/tags/errors.md>), [fabric](<https://devfeed.tech/tags/fabric.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>)

## AI overview

NVIDIA Attribution diagnoses distributed AI workload failures by correlating workload-scoped logs, metrics, telemetry, and scheduler state with deterministic rules. The Blueprint scopes a workload, collects observations, and calls the Attribution Service, which extracts and correlates events; the Blueprint returns a structured verdict with outcome, attributed domain, affected nodes, symptoms, and collector coverage. In the example, XID 149 on node-02 is identified as the root cause of a signal-integrity failure, while downstream NCCL errors are retained as evidence. When evidence supports multiple causes, the result can mark the sub-domain ambiguous; diagnosis quality depends on telemetry completeness. The self-hosted HTTP Service runs the core engine, while the customizable Python Blueprint integrates schedulers and data sources. Its quickstart uses Docker and uv with sample evidence. Rules ship with the Service and are not uploaded dynamically through its API.

## Source excerpt

In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...