# A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs

DevFeed: [A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs](<https://devfeed.tech/articles/a-cursed-bug-21940.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/a-cursed-bug/>)

Author: Nelson Elhage

Published: 2022-02-23T03:03:48Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [bug](<https://devfeed.tech/topics/bug.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [code](<https://devfeed.tech/tags/code.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [process](<https://devfeed.tech/tags/process.md>), [processes](<https://devfeed.tech/tags/processes.md>), [python](<https://devfeed.tech/tags/python.md>), [rdma](<https://devfeed.tech/tags/rdma.md>)

## AI overview

An Anthropic engineer describes a rare, nondeterministic bug in very large distributed training jobs. The bug originated in the RDMA software stack and caused subprocess launches to fail on a small fraction of nodes, sometimes producing Python exceptions or segmentation faults.

## Source excerpt

In my day job at Anthropic, we run relatively large distributed systems to train large language models. One of the joys of using a lot of computing resources, especially on somewhat niche software stacks, is that you spend a lot of time running into the long-tail of bugs which only happen rarely or in very unusual configurations, which you happen to be the first to encounter. These bugs are frustrating, but I also often enjoy them.