# Fault tolerant distributed training on Amazon EKS using NVRx

DevFeed: [Fault tolerant distributed training on Amazon EKS using NVRx](<https://devfeed.tech/articles/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx-31520.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx/>)

Author: Aravind Neelakantan

Published: 2026-09-16T18:59:25Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [NCCL](<https://devfeed.tech/topics/nccl.md>), [Amazon Elastic Kubernetes Service](<https://devfeed.tech/topics/amazon-elastic-kubernetes-service.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [amazon-elastic-kubernetes-service](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service.md>), [async](<https://devfeed.tech/tags/async.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [resiliency](<https://devfeed.tech/tags/resiliency.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

## AI overview

This tutorial integrates NVIDIA Resiliency Extension (NVRx) with PyTorch FSDP training on Amazon EKS. It covers asynchronous checkpointing, in-process restart, and in-job restart, and reports H100 benchmarks at 2- to 8-node scale with 99%+ training efficiency and recovery measured in seconds.

## Source excerpt

Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.