# distributed-training Published articles for distributed-training. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Fault tolerant distributed training on Amazon EKS using NVRx DevFeed: [Fault tolerant distributed training on Amazon EKS using NVRx]() Original publisher: [Read original article]() Author: Aravind Neelakantan Published: 2026-09-16T18:59:25Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [NCCL](), [Amazon Elastic Kubernetes Service](), [PyTorch](), [distributed-training](), [Resilience](), [GPU]() Tags: [advanced-300](), [amazon-eks](), [amazon-elastic-kubernetes-service](), [async](), [benchmarks](), [distributed-training](), [gpu](), [pytorch](), [resiliency](), [technical-how-to]() ### AI overview This tutorial integrates NVIDIA Resiliency Extension (NVRx) with PyTorch FSDP training on Amazon EKS. It covers asynchronous checkpointing, in-process restart, and in-job restart, and reports H100 benchmarks at 2- to 8-node scale with 99%+ training efficiency and recovery measured in seconds. ### Source excerpt Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery. ## Monitor TAS and gang scheduling for AI training in Kubernetes DevFeed: [Monitor TAS and gang scheduling for AI training in Kubernetes]() Original publisher: [Read original article]() Author: David Lentz; Kathy Lin Published: 2026-09-15T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Kubernetes](), [kueue](), [distributed-training](), [GPU](), [Monitoring](), [datadog]() Tags: [ai](), [batch](), [containers](), [datadog](), [distributed-training](), [gpu](), [gpu-monitoring](), [kubernetes](), [kueue](), [monitoring](), [scheduler]() ### AI overview This article explains why Kubernetes scheduling is insufficient for distributed AI training workloads and how topology-aware scheduling and gang scheduling address hardware placement and simultaneous startup requirements. It discusses implementing these capabilities with Kueue and the Coscheduling plugin, and monitoring and troubleshooting them with Datadog GPU Monitoring. ### Source excerpt Learn how Datadog helps you correlate Kueue, Coscheduling, GPU, and training framework signals to validate gang scheduling and topology-aware scheduling. ## Building a reliable cloud native foundation for distributed AI training DevFeed: [Building a reliable cloud native foundation for distributed AI training]() Original publisher: [Read original article]() Author: Abhi Kulkarni and Shishir Jindal, Atlassian Published: 2026-09-11T11:00:00Z Content type: article Language: en Sources: [Cloud Native Computing Foundation]() Topics: [Training AI Models](), [Machine learning](), [distributed-systems](), [GPU](), [cloud-infrastructure](), [Network design](), [Inference]() Tags: [ai-training](), [blog](), [distributed-training](), [gpu](), [infrastructure](), [storage]() ### AI overview The article explains how to make multi-node AI training reliable by treating inter-node communication, shared storage, hardware placement, network topology, and validation as platform concerns. It identifies RDMA for GPU-node communication and Lustre for concurrent training-data and checkpoint access. ### Source excerpt AI workloads are changing what platform teams need from infrastructure. Provisioning GPUs and standing up a cluster no longer makes a platform "AI-ready." Once training spans more than one node, the bottlenecks show up in places... ## Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI DevFeed: [Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI]() Original publisher: [Read original article]() Author: Dmytro Hryshchenko, Abhijeet Dhumal Published: 2026-09-09T03:32:28Z Content type: tutorial Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [Fine-tuning](), [voice ai](), [Training AI Models](), [Artificial Intelligence](), [distributed-training](), [GPU](), [lora](), [data](), [Open Source]() Tags: [ai](), [artificial-intelligence](), [distributed-training](), [fine-tuning](), [gpu](), [lora](), [open-source](), [speech](), [text-to-speech](), [training](), [voice](), [workflow]() ### AI overview This tutorial explains how to fine-tune the open source Orpheus-3B text-to-speech model for Turkish using Red Hat OpenShift AI and Kubeflow Trainer. It describes packaging distributed training in a TrainJob, scaling across nodes and GPUs, and using LoRA to keep memory usage below 16 GB. The reported result reduces speech errors by more than 90% compared with the base model. ### Source excerpt Can't Read, Won't Buy. That is the title CSA Research gave its survey of 8,709 consumers across 29 countries, and the numbers justify it: 76% prefer to buy in their own language, and 40% will never buy in another. The same rule governs what your product says out loud. The post Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI appeared first on Red Hat Developer. ## Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod DevFeed: [Pathway's brain-inspired architecture development on Amazon SageMaker HyperPod]() Original publisher: [Read original article]() Author: Paulo Aragão Published: 2026-09-08T19:12:51Z Content type: article Language: en Sources: [Artificial Intelligence]() Topics: [Language models](), [Training AI Models](), [compression and generalization](), [LLM evaluation / benchmarking]() Tags: [amazon](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [announcements](), [architecture](), [benchmark](), [chain-of-thought](), [distributed-training](), [intermediate-200](), [pytorch](), [transformer-architecture]() ### AI overview Pathway describes BDH, a brain-inspired architecture that performs reasoning in latent space rather than producing chain-of-thought token traces. The article covers its recurrent internal memory, its contrast with transformer limitations, and scaling training with Amazon SageMaker HyperPod. ### Source excerpt Pathway's Baby Dragon Hatchling (BDH) is a brain-inspired, post-transformer architecture that reasons in latent space instead of emitting chain-of-thought tokens. See how Pathway develops and scales BDH on Amazon SageMaker HyperPod, and how BDH-CQ set a new cost-efficiency mark on the ARC-AGI-1 benchmark. ## AI-Ready Private Cloud with Cisco and VMware DevFeed: [AI-Ready Private Cloud with Cisco and VMware]() Original publisher: [Read original article]() Author: sabina anja Published: 2026-09-08T15:33:39Z Content type: article Language: en Sources: [VMware Blogs]() Topics: [Artificial Intelligence](), [Network](), [Cloud](), [networking](), [cloud-infrastructure](), [distributed-training](), [datacenter](), [GPU](), [Latency](), [Inference](), [model-deployment](), [inference-endpoints]() Tags: [ai](), [cisco](), [cloud](), [cloud-infrastructure](), [cloud-platform](), [data-center](), [distributed-training](), [fabric](), [gpu](), [home-page](), [inference](), [inference-endpoints](), [latency](), [networking](), [object-storage](), [pipelines](), [private-cloud](), [telemetry](), [vcf-9-1](), [vcf-networking](), [vmware](), [vmware-cloud-foundation]() ### AI overview This article explains why an AI-ready private cloud requires more than adding GPUs. It focuses on how Broadcom and Cisco are integrating VMware Cloud Foundation with Cisco Nexus One Fabric to address AI workload networking, including bandwidth-intensive east-west traffic, bursty north-south traffic, latency, congestion management, and telemetry across virtual and physical infrastructure. ### Source excerpt An AI-ready private cloud is not simply a private cloud with GPUs added to it. What determines whether a private cloud platform can actually serve AI workloads effectively is everything built around them: how the fabric carries traffic, how the tenancy model lets teams consume capacity, and how policy and telemetry stay coherent across the ... Continued The post AI-Ready Private Cloud with Cisco and VMware appeared first on VMware Blogs. ## Kubeflow Has Graduated from CNCF DevFeed: [Kubeflow Has Graduated from CNCF]() Original publisher: [Read original article]() Author: Kubeflow Published: 2026-08-18T05:00:00Z Content type: release Language: en Sources: [Kubeflow]() Topics: [AI Infrastructure](), [AI Platform](), [Kubernetes](), [Cloud Native Ecosystem](), [distributed-training](), [Fine-tuning](), [model-serving]() Tags: [ai-infrastructure](), [announcement](), [cloud-native](), [cncf](), [distributed-training](), [fine-tuning](), [inference](), [kubernetes](), [model-serving](), [production]() ### AI overview Kubeflow has graduated from the Cloud Native Computing Foundation, recognizing its maturity and adoption as a Kubernetes-native platform for AI and machine learning workloads. The article highlights its community growth, enterprise use, security audit, governance work, and focus on scalable, portable infrastructure. ### Source excerpt Kubeflow is a CNCF Graduated Project ## How to Choose Full-Stack Observability for NVIDIA AI Factories DevFeed: [How to Choose Full-Stack Observability for NVIDIA AI Factories]() Original publisher: [Read original article]() Author: Jorge Cardoso Published: 2026-08-12T16:13:47Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [observability](), [AI Infrastructure](), [NVIDIA DGX](), [InfiniBand](), [telemetry](), [dashboards](), [NCCL](), [Monitoring]() Tags: [ai-factory](), [ai-infrastructure](), [data-center-cloud](), [distributed-training](), [featured](), [infiniband](), [infrastructure](), [monitoring](), [nccl](), [networking](), [networking-communications](), [nvidia-dgx](), [observability](), [operations](), [orchestration](), [performance](), [storage](), [systems](), [telemetry]() ### AI overview A practical guide to choosing a full-stack observability strategy for NVIDIA AI infrastructure. It explains how to connect telemetry across compute, networking, storage, orchestration, and applications, using an InfiniBand gray-failure example to show how degraded hardware and NCCL collective-operation delays can reduce distributed-training throughput. ### Source excerpt AI infrastructure spans multiple layers, from compute and networking to storage, orchestration, and applications. When performance degrades, identifying the... ## NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure DevFeed: [NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-30T16:00:00Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [debugging](), [Processes]() Tags: [ai](), [ai-infrastructure](), [benchmarks](), [blackwell](), [cloud](), [cloud-services](), [cpu](), [data-center-cloud](), [debugging](), [developer-tools-techniques](), [dgx-cloud](), [diagnostics](), [distributed-training](), [featured](), [gb200](), [gb300-nvl72](), [grace-cpu](), [hardware](), [hopper](), [infiniband](), [infrastructure](), [kernel](), [linux](), [nvidia](), [nvl72](), [performance](), [training]() ### AI overview A troubleshooting guide for closing AI-training throughput gaps between NVIDIA reference architectures and partner clusters. It covers configuration and installation issues across memory management, CPU power and NUMA placement, NCCL queue-pair concurrency, and hardware setup. ### Source excerpt Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We... ## KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience DevFeed: [KubeCon + CloudNativeCon India 2026: Our Kubeflow Community Experience]() Original publisher: [Read original article]() Author: Khushi Agrawal Published: 2026-07-27T05:00:00Z Content type: article Language: en Sources: [Kubeflow]() Topics: [Kubernetes](), [MLOps](), [distributed-training](), [model-serving](), [Large Language Model](), [Retrieval Augmented Generation (RAG)](), [GPU](), [Cloud]() Tags: [2026](), [ai](), [cloud-native](), [community](), [distributed-training](), [gpu](), [india](), [kubecon](), [kubernetes](), [llms](), [machine-learning](), [mlops](), [model-serving](), [model-training](), [rag]() ### AI overview A Kubeflow community recap of KubeCon + CloudNativeCon India 2026 in Mumbai. It describes Kubeflow's presence at the CNCF Project Pavilion, common architecture questions, featured talks, booth discussions about production machine learning workflows, distributed training, model serving, GPU resource allocation, LLMs, and RAG pipelines, and efforts to onboard contributors. ### Source excerpt Introduction ## Achieving Near-Linear Training Scalability for Pinterest's Foundation Models DevFeed: [Achieving Near-Linear Training Scalability for Pinterest's Foundation Models]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-25T16:01:02Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Training AI Models](), [NCCL](), [networking]() Tags: [aws](), [distributed-training](), [embedding](), [engineering](), [foundation-models](), [gpu](), [machine-learning](), [nccl](), [pinterest](), [scalability](), [training]() ### AI overview Pinterest describes optimizing multi-node distributed training for an embedding-heavy foundation ranking model. The reported improvements raise scaling from 1.13x to 2.0x at two nodes and from 1.21x to 3.9x at four nodes, with 7.5x scaling at eight nodes. ### Source excerpt Sheng Huang | Software Engineer, AI Platform; Pong Eksombatchai | Machine Learning Engineer, Applied Sciences; Saurabh Vishwas Joshi | Software Engineer, AI Platform; Gaurav Arora | Software Engineer, AI Platform; Karthik Anantha Padmanabhan | Engineering Director, AI Platform At Pinterest, foundation models power recommendations for over 600 million monthly active users. Our latest Foundation Model (ACM RecSys 2025) pre-trains on two years of user activity data and is deployed into Home feed and Related Pins ranking, the platform's two most important recommendation systems. Multi-node distributed training is the key to unlocking the next level of that capacity.¹ But when we first attempted multi-node training, adding a second machine made training 5x slower, producing a scaling factor of roughly 0.2x. Enabling AWS Elastic Fabric Adapter (EFA) for OS-bypass networking fixed the networking layer and recovered a viable baseline, but scaling was still poor: 1.13x at 2 nodes and 1.21x at 4 nodes. Three extra nodes, 3x more GPUs, 3x more cost, yet only 21% more throughput. This post describes how we took 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x (97.5% of ideal), then extended to 8 nodes at 7.5x. The larger models this unlocked have driven significant engagement gains across Pinterest's recommendation surfaces. Figure 1: Training scalability before and after optimization. Left: before EFA and optimization, adding a second node degraded throughput to 0.2x of single-node. Right: after optimization, scaling is near-linear across 2, 4, and 8 nodes, with 8-node reaching 7.5x (93.75% of ideal).Background Training scalability measures whether adding more resources yields proportionally more throughput. Training efficiency measures how much throughput you extract from the same resources. This post focuses on scalability. Our Foundation Ranking Model is embedding-heavy: approximately 99% of parameters reside in embedding tables, with the dense transf ## Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel DevFeed: [Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel]() Original publisher: [Read original article]() Author: Adil Asif; Alexandros Koumparoulis; Wenwen Gao; Sylendran Arunagiri; David Messina; Bernard Nguyen Published: 2026-06-24T16:00:13Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [NeMo](), [Fine-tuning](), [AI Models](), [Mixture of Experts (MoE)](), [distributed-training](), [Generative AI](), [Artificial Intelligence](), [GPU](), [PyTorch](), [vllm](), [sglang]() Tags: [ai](), [ai-models](), [building](), [compute](), [distributed-training](), [fine-tuning](), [framework](), [generative](), [generative-ai](), [gpu](), [infrastructure](), [kernels](), [memory](), [model](), [moe](), [nemo](), [nemotron](), [nvidia](), [open](), [transformers]() ### AI overview This article explains how NVIDIA NeMo AutoModel accelerates fine-tuning of mixture-of-experts models by extending Transformers v5 with Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels. It describes API compatibility, distributed execution, dynamic weight loading, and reported gains of 3.4-3.7x higher training throughput and 29-32% lower GPU memory use. ### Source excerpt NVIDIA NeMo AutoModel is an open library part of the NVIDIA NeMo framework for building custom generative AI models at scale. NeMo AutoModel builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5's dynamic weight loading to bring those optimizations to a broad and growing set of model families. ## Building Blocks for Foundation Model Training and Inference on AWS DevFeed: [Building Blocks for Foundation Model Training and Inference on AWS]() Original publisher: [Read original article]() Author: Keita Watanabe; Pavel Belevich; Aman Shanbhag Published: 2026-05-11T23:18:26Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [foundation-models](), [Amazon Web Services](), [Training AI Models](), [Inference](), [distributed-training](), [cloud-infrastructure](), [Orchestration](), [PyTorch](), [Kubernetes](), [Grafana](), [Prometheus](), [Monitoring]() Tags: [ai](), [aws](), [building](), [distributed-training](), [frameworks](), [grafana](), [hardware](), [inference](), [infrastructure](), [kubernetes](), [latency](), [low-latency](), [model](), [model-development](), [monitoring](), [networking](), [observability](), [open-source](), [prometheus](), [training]() ### AI overview An introductory technical article about the infrastructure and open-source software building blocks required for foundation-model pre-training, post-training, and inference on AWS. It discusses accelerator compute, low-latency networking, distributed storage, orchestration, ML frameworks, and observability tools. ### Source excerpt Figure: Adapted from "AI's Three Scaling Laws, Explained" (NVIDIA Blog). Taken together, these scaling regimes push the foundation-model lifecycle--pre-training, post-training, and inference--toward convergent infrastructure requirements: tightly coupled accelerator compute, a high-bandwidth low-latency network, and a distributed storage backend. ## Kubeflow Trainer v2.2: JAX & XGBoost Runtimes, Flux for HPC Support, and TrainJob progress and metrics observability DevFeed: [Kubeflow Trainer v2.2: JAX & XGBoost Runtimes, Flux for HPC Support, and TrainJob progress and metrics observability]() Original publisher: [Read original article]() Author: Kubeflow Trainer Team Published: 2026-03-20T05:00:00Z Content type: release Language: en Sources: [Kubeflow]() Topics: [Kubernetes](), [distributed-training](), [observability](), [flux](), [Fine-tuning]() Tags: [distributed-training](), [fine-tuning](), [flux](), [gpu](), [hugging-face](), [kubernetes](), [observability](), [release](), [trainer]() ### AI overview Kubeflow Trainer v2.2 adds native JAX and XGBoost training runtimes, Flux runtime support for HPC workloads, enhanced training-job observability, timeout policies, and more flexible runtime configuration through new APIs. It also unifies supported training workloads under the TrainJob abstraction and enables distributed JAX workloads on Kubernetes. ### Source excerpt Just a little over one week ahead of KubeCon + CloudNativeCon EU 2026, the Kubeflow team is excited to ship Trainer v2.2. The v2.2 release reinforces our commitment to expanding the Kubeflow Trainer ecosystem - meeting developers where they are by adding native support for JAX, XGBoost, and Flux, while also delivering deeper observability into training jobs. ## Introducing Storage Buckets on the Hugging Face Hub DevFeed: [Introducing Storage Buckets on the Hugging Face Hub]() Original publisher: [Read original article]() Author: Lucain Pouget; Eliott Coyac; Adrien Carreira; Victor Mustar; Julien Chaumond; Quentin Lhoest; Pierric Cistac; Sylvestre Bcht; Hugo Larcher; Rajat Arya Published: 2026-03-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [hugging face](), [xet](), [datasets](), [distributed-training](), [Machine learning](), [cloud-infrastructure](), [Open Source]() Tags: [announcement](), [artificial-intelligence](), [cloud](), [datasets](), [distributed-training](), [hub](), [hugging-face](), [open-source](), [storage](), [training](), [xet]() ### AI overview The article introduces Storage Buckets on the Hugging Face Hub, a mutable S3-like storage option for machine-learning artifacts. Built on Xet, Buckets use chunking and deduplication to reduce redundant transfers, storage use, and enterprise billing, while pre-warming places frequently accessed data near distributed-training compute. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries DevFeed: [Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Lewis Tunstall; Edward Beeching; Albert Villanova del Moral; Nouamane Tazi; Leandro von Werra; Sergio Paniego Published: 2026-03-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [distributed-training](), [Reinforcement learning](), [distributed-systems](), [Orchestration](), [NCCL](), [post-training](), [lora](), [Mixture of Experts (MoE)](), [grpo](), [Chain-of-thought](), [Open Source Models & Datasets]() Tags: [agentic](), [analysis](), [chain-of-thought](), [distributed-training](), [grpo](), [llm](), [lora](), [mixture-of-experts](), [nccl](), [open-source](), [post-training](), [rl](), [tokens]() ### AI overview This article surveys 16 open-source libraries for asynchronous reinforcement-learning training. It explains how separating inference and training across GPU pools, using rollout buffers, and synchronizing weights asynchronously can reduce training-GPU idle time. The comparison covers orchestration, buffering, weight synchronization, staleness management, partial rollouts, LoRA, and distributed-training backends, highlighting Ray, NCCL broadcasts, limited LoRA support, and distributed MoE as an emerging differentiator. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Ulysses Sequence Parallelism: Training with Million-Token Contexts DevFeed: [Ulysses Sequence Parallelism: Training with Million-Token Contexts]() Original publisher: [Read original article]() Author: Kashif Rasul; Stas Bekman Published: 2026-03-09T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Artificial Intelligence](), [long-context](), [GPU](), [hugging face](), [Open Source](), [trl]() Tags: [accelerate](), [ai](), [artificial-intelligence](), [distributed-training](), [ecosystem](), [gpu](), [guide](), [long-context](), [open-source](), [tokens](), [transformers](), [trl]() ### AI overview This article explains Ulysses Sequence Parallelism, a method for training transformer models with very long or million-token contexts by sharding sequences and partitioning attention heads across multiple GPUs. It describes the all-to-all communication steps and integration across the Hugging Face ecosystem, including Accelerate, Transformers Trainer, and TRL's SFTTrainer. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## How Shopify uses SkyPilot to route machine-learning workloads across multi-cloud GPU clusters DevFeed: [How Shopify uses SkyPilot to route machine-learning workloads across multi-cloud GPU clusters]() Original publisher: [Read original article]() Author: Javier Moreno Published: 2026-01-26T14:49:55Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [skypilot](), [Cloud](), [Machine learning](), [InfiniBand]() Tags: [cloud](), [development](), [distributed-training](), [infiniband](), [kubernetes](), [kubernetes-clusters](), [multi-cloud](), [open-source](), [shopify](), [skypilot](), [yaml]() ### AI overview Shopify describes using SkyPilot to run machine-learning workloads across existing Kubernetes clusters on multiple clouds. A custom plugin routes jobs based on requested hardware and workload needs, while supporting multi-team management, cost tracking, fair scheduling, and policy enforcement. ### Source excerpt GPUs are annoying. Shopify uses SkyPilot to make them less so: one YAML file, multiple clouds, clean development ergonomics. ## Differentially private machine learning at scale with JAX-Privacy DevFeed: [Differentially private machine learning at scale with JAX-Privacy]() Original publisher: [Read original article]() Published: 2025-11-12T15:32:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Machine Learning & Artificial Intelligence](), [Algorithms, Complexity](), [Deep learning](), [distributed-training](), [data](), [Google](), [High-Performance Computing](), [Programming]() Tags: [algorithms](), [algorithms-theory](), [data](), [deep-learning](), [distributed-training](), [google](), [high-performance-computing](), [libraries](), [machine-learning](), [release](), [research](), [responsible-ai](), [scalability](), [security-privacy-and-abuse-prevention](), [training]() ### AI overview Google announces JAX-Privacy 1.0, a library for differentially private machine learning built on JAX. The release is intended to help researchers and developers implement, audit, and scale private training workflows for deep learning models using large datasets and distributed training. ### Source excerpt Algorithms & Theory ## How Deliveroo Built Its Machine Learning Platform DevFeed: [How Deliveroo Built Its Machine Learning Platform]() Original publisher: [Read original article]() Author: Saikrishna Desaraju Published: 2025-07-02T00:00:00Z Content type: article Language: en Sources: [Deliveroo]() Topics: [Machine Learning & Artificial Intelligence](), [Deployment](), [model-deployment](), [Kubernetes](), [Amazon Web Services](), [distributed-training](), [Inference](), [Python](), [PyTorch](), [Tensorflow]() Tags: [aws](), [deployment](), [distributed-training](), [inference](), [kubernetes](), [machine-learning](), [machine-learning-platform](), [python](), [pytorch](), [rest](), [tensorflow]() ### AI overview Deliveroo describes its machine learning platform for standardizing model-building and deployment workflows. The platform combines open-source tools, in-house components, Kubernetes on AWS EKS, TensorFlow, PyTorch, Metaflow, and Argo; it also supports GPU and distributed training and includes Inferoo for real-time model predictions. ### Source excerpt Enter Deliveroo's ML Platform For the past three years, we have been building Deliveroo's Machine Learning Platform, or the ML Platform as we like to call it. The ML Platform boosts our model-building and deployment capabilities by standardising ML workflows, streamlining the end-to-end development process and simplifying model deployment. Besides saving software engineering effort through centralising tooling, the ML Platform also reduces the time that our ML engineers spend on infrastructure tasks. As a result, our ML engineers can now iterate their ML models 2-3x faster than before. What makes our ML Platform tick? At the core of our ML Platform lies a carefully curated tech stack - an integrated suite of infrastructure tools, services, and libraries. It blends robust open source technologies with purpose-built, in-house components. Key open source tools include Kubernetes, Argo, and Metaflow, all seamlessly connected with leading ML frameworks like TensorFlow and PyTorch. We choose mature, community-driven solutions and actively contribute back where we can. This entire ecosystem is powered by AWS, running on EKS, and anchored by our data warehouse. In the sections that follow, we'll take a closer look at the key components that drive our ML Platform. Metaflow To give ML engineers seamless access to scalable compute on Kubernetes, we use Metaflow, a powerful Python library that helps break down complex model-building workflows into smaller, manageable jobs. These jobs are orchestrated by Argo, one of the backbone tools in our infrastructure. One of Metaflow's biggest strengths is its flexibility. It allows engineers to move effortlessly between local development, staging on Kubernetes, and production deployment--helping teams iterate quickly as they experiment and scale. Recently, we added support for GPUs and distributed training, enabling faster training times and the ability to work with much larger datasets. Inferoo For models that require real-time predictio ## Mini-R1: Reproduce Deepseek R1 "aha moment" a RL tutorial DevFeed: [Mini-R1: Reproduce Deepseek R1 "aha moment" a RL tutorial]() Original publisher: [Read original article]() Author: Philipp Schmid Published: 2025-01-31T10:29:40Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [deepseek](), [grpo](), [Reinforcement learning](), [distributed-training](), [Large Language Model](), [vllm](), [NCCL](), [Nvidia](), [qwen](), [OpenAI]() Tags: [blog](), [blog-post](), [deepseek](), [development](), [distributed-training](), [grpo](), [llms](), [model](), [nvidia](), [open](), [qwen](), [reasoning](), [reinforcement-learning](), [rl](), [train](), [training](), [tutorial](), [vllm]() ### AI overview This tutorial reproduces a small version of DeepSeek-R1's reported reinforcement-learning "aha moment" by training an open model with Group Relative Policy Optimization (GRPO) to solve the Countdown numbers game. It covers self-verification, search behavior, distributed training with DeepSpeed and vLLM, and a four-NVIDIA-H100-GPU setup. ### Source excerpt The release of Deepseek R1 shocked the industry. Why? Well, DeepSeek-R1 is an open model that rivals OpenAI's o1 in complex reasoning tasks, introduced using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach. They not only released the model, but also a research paper on how they did it. In the paper they described an "aha moment" when using pure RL to train the model. ## Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations DevFeed: [Implementing Multi-GPU Distributed Training for Stitch Fix's Personalized Recommendations]() Original publisher: [Read original article]() Published: 2023-06-08T09:00:00Z Content type: article Language: en Sources: [Stitch Fix]() Topics: [distributed-training](), [recommendations](), [GPU](), [sharding](), [Training AI Models](), [Multi-GPU](), [PyTorch]() Tags: [distributed-training](), [gpu](), [model-training](), [multi-gpu](), [parallel](), [performance](), [pytorch](), [recommendations](), [sharding]() ### AI overview This Stitch Fix engineering article explains how the company implemented multi-GPU distributed training for its Client Time Series Model (CTSM), a PyTorch-based model used in personalized recommendations. It describes sharding training data across GPUs and training mini-batches in parallel to reduce training time, along with the surrounding retraining and deployment workflow. ### Source excerpt Stitch Fix uses a cutting-edge multi-tiered recommender system stack to personalize styling recommendations at scale. This stack comprises several critical components, including feature generation, scoring, ranking, and inventory optimization techniques. Our scoring module is based on the Client Time Series Model (CTSM) which is an award winning novel sequence based model that uses temporally masked encoders. CTSM is built using PyTorch, and was initially trained on a single Graphics Processing Unit (GPU) instance. Since we first put this model into production last year, we have launched several updates to the model that improved its performance. Many of these improvements involved adding new features or increasing the time window of our training data. As a result, the model training time increased significantly, making it harder for us to iterate quickly and get feedback on new ideas we want to try for improving the model. We needed a way to reduce the model training time. This blog delves into the steps we followed to overcome this challenge and our journey to implement multi-GPU distributed model training for CTSM. By sharding the training data across multiple GPUs and training multiple mini-batches in parallel, we aimed to achieve significant reductions in training time. We present empirical results showcasing the observed reduction in training time when we scaled up resources from 1 to N GPUs, and share some future directions we are considering in our continued effort to speed up model training. Model Training Workflow The scores generated by CTSM are leveraged by multiple downstream services to get insight into what items a client is likely to purchase. The model is retrained at a regular cadence to ensure that it is using the most updated information about each client when making predictions and does not degrade in its performance. We leverage configuration driven machine learning pipelines to set up a Directed Acyclic Graph (DAG) that automatically retrains ## A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs DevFeed: [A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs]() Original publisher: [Read original article]() Author: Nelson Elhage Published: 2022-02-23T03:03:48Z Content type: opinion Language: en Sources: [Nelson Elhage]() Topics: [bug](), [distributed-systems](), [distributed-training](), [Processes](), [coding]() Tags: [bug](), [code](), [debugging](), [distributed-systems](), [distributed-training](), [process](), [processes](), [python](), [rdma]() ### AI overview An Anthropic engineer describes a rare, nondeterministic bug in very large distributed training jobs. The bug originated in the RDMA software stack and caused subprocess launches to fail on a small fraction of nodes, sometimes producing Python exceptions or segmentation faults. ### Source excerpt In my day job at Anthropic, we run relatively large distributed systems to train large language models. One of the joys of using a lot of computing resources, especially on somewhat niche software stacks, is that you spend a lot of time running into the long-tail of bugs which only happen rarely or in very unusual configurations, which you happen to be the first to encounter. These bugs are frustrating, but I also often enjoy them.