# NCCL Published articles for NCCL. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL DevFeed: [Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Sergio Paniego Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [lora](), [grpo](), [jobs](), [vllm](), [NCCL](), [async](), [hugging face]() Tags: [async](), [grpo](), [hub](), [hugging-face](), [jobs](), [llm](), [lora](), [nccl](), [open-source](), [rl](), [storage](), [trl](), [vllm]() ### AI overview This article describes asynchronous GRPO training with a LoRA adapter across separate Hugging Face Jobs. The adapter is synchronized to vLLM replicas through a shared Storage Bucket, while a proxy handles authentication, rollout routing, and adapter-load broadcasts. Five runs reduced the time for 500 steps from 3 hours 27 minutes to 53 minutes. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo DevFeed: [Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-25T20:57:54Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Large Language Model](), [Inference](), [model-deployment](), [Nvidia](), [Optimization](), [CUDA](), [Processes](), [GPU](), [NCCL](), [sglang]() Tags: [agentic-ai-generative-ai](), [cuda](), [cuda-x](), [data-science](), [deployment](), [developer-tools-techniques](), [gpu](), [inference](), [llm](), [models](), [nccl](), [nvidia](), [optimization](), [process]() ### AI overview This article explains NVIDIA Dynamo's shadow engine recovery for LLM inference. A fully initialized shadow engine shares weights with the active engine and takes over after a process failure, reducing recovery from a 283-second cold restart to 7.3 seconds in the described two-worker deployment. ### Source excerpt When an LLM engine process fails, the standard recovery path involves a cold restart. This requires loading weights into HBM from storage, compiling kernels,... ## How to Choose Full-Stack Observability for NVIDIA AI Factories DevFeed: [How to Choose Full-Stack Observability for NVIDIA AI Factories]() Original publisher: [Read original article]() Author: Jorge Cardoso Published: 2026-08-12T16:13:47Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [observability](), [AI Infrastructure](), [NVIDIA DGX](), [InfiniBand](), [telemetry](), [dashboards](), [NCCL](), [Monitoring]() Tags: [ai-factory](), [ai-infrastructure](), [data-center-cloud](), [distributed-training](), [featured](), [infiniband](), [infrastructure](), [monitoring](), [nccl](), [networking](), [networking-communications](), [nvidia-dgx](), [observability](), [operations](), [orchestration](), [performance](), [storage](), [systems](), [telemetry]() ### AI overview A practical guide to choosing a full-stack observability strategy for NVIDIA AI infrastructure. It explains how to connect telemetry across compute, networking, storage, orchestration, and applications, using an InfiniBand gray-failure example to show how degraded hardware and NCCL collective-operation delays can reduce distributed-training throughput. ### Source excerpt AI infrastructure spans multiple layers, from compute and networking to storage, orchestration, and applications. When performance degrades, identifying the... ## Achieving Near-Linear Training Scalability for Pinterest's Foundation Models DevFeed: [Achieving Near-Linear Training Scalability for Pinterest's Foundation Models]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-25T16:01:02Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Training AI Models](), [NCCL](), [networking]() Tags: [aws](), [distributed-training](), [embedding](), [engineering](), [foundation-models](), [gpu](), [machine-learning](), [nccl](), [pinterest](), [scalability](), [training]() ### AI overview Pinterest describes optimizing multi-node distributed training for an embedding-heavy foundation ranking model. The reported improvements raise scaling from 1.13x to 2.0x at two nodes and from 1.21x to 3.9x at four nodes, with 7.5x scaling at eight nodes. ### Source excerpt Sheng Huang | Software Engineer, AI Platform; Pong Eksombatchai | Machine Learning Engineer, Applied Sciences; Saurabh Vishwas Joshi | Software Engineer, AI Platform; Gaurav Arora | Software Engineer, AI Platform; Karthik Anantha Padmanabhan | Engineering Director, AI Platform At Pinterest, foundation models power recommendations for over 600 million monthly active users. Our latest Foundation Model (ACM RecSys 2025) pre-trains on two years of user activity data and is deployed into Home feed and Related Pins ranking, the platform's two most important recommendation systems. Multi-node distributed training is the key to unlocking the next level of that capacity.¹ But when we first attempted multi-node training, adding a second machine made training 5x slower, producing a scaling factor of roughly 0.2x. Enabling AWS Elastic Fabric Adapter (EFA) for OS-bypass networking fixed the networking layer and recovered a viable baseline, but scaling was still poor: 1.13x at 2 nodes and 1.21x at 4 nodes. Three extra nodes, 3x more GPUs, 3x more cost, yet only 21% more throughput. This post describes how we took 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x (97.5% of ideal), then extended to 8 nodes at 7.5x. The larger models this unlocked have driven significant engagement gains across Pinterest's recommendation surfaces. Figure 1: Training scalability before and after optimization. Left: before EFA and optimization, adding a second node degraded throughput to 0.2x of single-node. Right: after optimization, scaling is near-linear across 2, 4, and 8 nodes, with 8-node reaching 7.5x (93.75% of ideal).Background Training scalability measures whether adding more resources yields proportionally more throughput. Training efficiency measures how much throughput you extract from the same resources. This post focuses on scalability. Our Foundation Ranking Model is embedding-heavy: approximately 99% of parameters reside in embedding tables, with the dense transf ## Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL DevFeed: [Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Lewis Tunstall; Edward Beeching; Albert Villanova del Moral; Leandro von Werra; Sergio Paniego Published: 2026-05-27T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [AI, ML & Data Engineering](), [NCCL]() Tags: [artificial-intelligence](), [compute](), [hub](), [hugging-face](), [inference](), [llm](), [model](), [models](), [nccl](), [open-source](), [payload](), [policy](), [rl](), [space](), [storage](), [sync](), [train](), [training](), [trl](), [update](), [vllm]() ### AI overview The article describes delta weight synchronization for asynchronous reinforcement-learning training. A TRL change stores only modified model weights in sparse safetensors files and lets vLLM fetch them from a Hugging Face bucket, reducing transfer payloads and enabling disaggregated training without a shared cluster. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Running the Latest vLLM on the NVIDIA DGX Spark DevFeed: [Running the Latest vLLM on the NVIDIA DGX Spark]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-21T13:00:00Z Content type: tutorial Language: en Sources: [Techno Tim]() Topics: [vllm](), [DGX Spark](), [Docker](), [Dockerfile](), [GitHub Actions](), [NCCL](), [CUDA](), [PyTorch](), [GitHub](), [NGC](), [GPU](), [Ubuntu]() Tags: [ai](), [cuda](), [dgx-spark](), [docker](), [docker-image](), [github](), [github-actions](), [gpu](), [homelab](), [nccl](), [ngc](), [nvidia-dgx](), [pytorch](), [self-hosted](), [ubuntu](), [vllm]() ### AI overview This tutorial presents a reproducible Docker image pipeline for running current vLLM releases on the NVIDIA DGX Spark's GB10 ARM64 platform. It explains why NVIDIA's NGC images can lag upstream and describes a GitHub Actions build on a self-hosted Spark runner with pinned inputs and generated lockfiles. ### Source excerpt When I built my local AI cluster on a pair of ASUS Ascent GX10s, the hard part was not serving a model. The hard part was getting a working vLLM image with current components. NVIDIA's official image was already over a month behind by the time I needed it, and waiting on their release schedule was not an option. If you saw that post, you know the GX10 is an ARM64 machine built around NVIDIA's ... ## I Built a 256GB Local AI Cluster on My Desk DevFeed: [I Built a 256GB Local AI Cluster on My Desk]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-18T13:00:00Z Content type: article Language: en Sources: [Techno Tim]() Topics: [Local AI](), [Artificial Intelligence](), [agentic-coding](), [Hardware](), [Homelab](), [coding](), [model-serving](), [vllm](), [networking](), [Docker](), [NCCL](), [Monitoring]() Tags: [agentic-coding](), [ai](), [docker](), [github](), [hardware](), [homelab](), [local-ai](), [model-serving](), [monitoring](), [nccl](), [networking](), [self-hosted](), [self-hosted-ai](), [server](), [vllm]() ### AI overview The article tests a two-node local AI cluster built from ASUS Ascent GX10 systems, using one machine for practical agentic coding and two machines to run a larger local model. It evaluates a real coding workflow involving model serving, storage, networking, memory, Docker, NCCL, monitoring, power draw, and the application being built, concluding that local AI is capable and cloud-independent but still operationally complex. ### Source excerpt I have been covering local and self-hosted AI for a few years now - from running models privately at home to what is still running in my homelab today. But to run the larger, more capable models, you need something more specialized than a general-purpose home server. I wanted to know how good local AI has actually gotten, so I built a mini AI cluster on my desk and used it for a real coding wo... ## Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries DevFeed: [Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Lewis Tunstall; Edward Beeching; Albert Villanova del Moral; Nouamane Tazi; Leandro von Werra; Sergio Paniego Published: 2026-03-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [distributed-training](), [Reinforcement learning](), [distributed-systems](), [Orchestration](), [NCCL](), [post-training](), [lora](), [Mixture of Experts (MoE)](), [grpo](), [Chain-of-thought](), [Open Source Models & Datasets]() Tags: [agentic](), [analysis](), [chain-of-thought](), [distributed-training](), [grpo](), [llm](), [lora](), [mixture-of-experts](), [nccl](), [open-source](), [post-training](), [rl](), [tokens]() ### AI overview This article surveys 16 open-source libraries for asynchronous reinforcement-learning training. It explains how separating inference and training across GPU pools, using rollout buffers, and synchronizing weights asynchronously can reduce training-GPU idle time. The comparison covers orchestration, buffering, weight synchronization, staleness management, partial rollouts, LoRA, and distributed-training backends, highlighting Ray, NCCL broadcasts, limited LoRA support, and distributed MoE as an emerging differentiator. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## What's new in TensorFlow 2.18 DevFeed: [What's new in TensorFlow 2.18]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2024-10-28T19:00:00Z Content type: release Language: en Sources: [The TensorFlow Blog]() Topics: [Tensorflow](), [releases](), [LiteRT](), [NumPy](), [CUDA](), [cudnn](), [NCCL](), [Google]() Tags: [announcement](), [cuda](), [cudnn](), [gpu](), [litert](), [migration](), [nccl](), [nvidia](), [nvidia-rtx](), [python](), [release](), [releases](), [reproducible-builds](), [tensorflow](), [tensorflow-core]() ### AI overview TensorFlow 2.18 introduces NumPy 2.0 compatibility updates, transitions TFLite development to the LiteRT repository, and adds hermetic CUDA, cuDNN, and NCCL dependencies for more reproducible source builds. Binary packages add dedicated kernels for compute capability 8.9 GPUs, including NVIDIA RTX 40 series, L4, and L40, while dropping precompiled support for Maxwell GPUs. ### Source excerpt Posted by the TensorFlow team TensorFlow 2.18 has been released! Highlights of this release (and 2.17) include NumPy 2.0, LiteRT repository, CUDA Update, Hermetic CUDA and more. For the full release notes, please click here. Note: Release updates on the new multi-backend Keras will be published on keras.io, starting with Keras 3.0. For more information, please see https://keras.io/keras_3/. TensorFlow Core NumPy 2.0 The upcoming TensorFlow 2.18 release will include support for NumPy 2.0. While the majority of TensorFlow APIs will function seamlessly with NumPy 2.0, this may break some edge cases of usage, e.g., out-of-boundary conversion errors and numpy scalar representation errors. You can consult the following common solutions. Note that NumPy's type promotion rules have been changed (See NEP 50 for details). This may change the precision at which computations happen, leading either to type errors or to numerical changes to results. Please see the NumPy 2 migration guide. We've updated some TensorFlow tensor APIs to maintain compatibility with NumPy 2.0 while preserving the out-of-boundary conversion behavior in NumPy 1.x. LiteRT Repository We're making some changes to how LiteRT (formerly known as TFLite) is developed. Over the coming months, we'll be gradually transitioning TFLite's codebase to LiteRT. Once the migration is complete, we'll start accepting contributions directly through the LiteRT repository. There will no longer be any binary TFLite releases and developers should switch to LiteRT for the latest updates. Hermetic CUDA If you build TensorFlow from source, Bazel will now download specific versions of CUDA, CUDNN and NCCL distributions, and then use those tools as dependencies in various Bazel targets. This enables more reproducible builds for Google ML projects and supported CUDA versions because the build no longer relies on the locally installed versions. More details are provided here. CUDA Update TensorFlow binary distributions now ship with d ## Distributed Fast Fourier Transform in TensorFlow DevFeed: [Distributed Fast Fourier Transform in TensorFlow]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2023-08-24T17:00:00Z Content type: tutorial Language: en Sources: [The TensorFlow Blog]() Topics: [AI, ML & Data Engineering](), [GPU](), [NCCL]() Tags: [api](), [gpu](), [machine-learning](), [nccl](), [performance](), [tensorflow](), [tensorflow-core]() ### AI overview TensorFlow v2 adds native Distributed FFT support through DTensor. The article explains how sharded tensors can use existing FFT operations across multiple devices, and reports that communication and data shuffling dominate runtime in the demonstrated setup. ### Source excerpt Posted by Ruijiao Sun, Google Intern - DTensor team Fast Fourier Transform is an important method of signal processing, which is commonly used in a number of ways, including speeding up convolutions, extracting features, and regularizing models. Distributed Fast Fourier Transform (Distributed FFT) offers a way to compute Fourier Transforms in models that work with image-like datasets that are too large to fit into the memory of a single accelerator device. In a previous Google Research Paper, "Large-Scale Discrete Fourier Transform on TPUs" by Tianjian Lu, a Distributed FFT algorithm was implemented for TensorFlow v1 as a library. This work presents the newly added native support in TensorFlow v2 for Distributed FFT, through the new TensorFlow distribution API, DTensor. About DTensor DTensor is an extension to TensorFlow for synchronous distributed computing. It distributes the program and tensors through a procedure called Single program, multiple data (SPMD) extension. DTensor offers an uniform API for traditional data and model parallelism patterns used widely in Machine Learning. Example Usage The API interface for distributed FFT is the same as the original FFT in TensorFlow. Users just need to pass a sharded tensor as an input to the existing FFT ops in TensorFlow, such as tf.signal.fft2d. The output of a distributed FFT becomes sharded too. import TensorFlow as tf from TensorFlow.experimental import dtensor # Set up devices device_type = dtensor.preferred_device_type() if device_type == 'CPU': cpu = tf.config.list_physical_devices(device_type) tf.config.set_logical_device_configuration(cpu[0], [tf.config.LogicalDeviceConfiguration()] * 8) if device_type == 'GPU': gpu = tf.config.list_physical_devices(device_type) tf.config.set_logical_device_configuration(gpu[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1000)] * 8) dtensor.initialize_accelerator_system() # Create a mesh mesh = dtensor.create_distributed_mesh(mesh_dims=[('x', 1), ('y', 2), ('z', 4)],