# Multi-GPU

Multi-GPU programming enables applications to use multiple GPUs concurrently for greater aggregate performance, memory capacity, and bandwidth.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Multilevel Graph Partitioning on GPU Supercomputers

DevFeed: [Multilevel Graph Partitioning on GPU Supercomputers](<https://devfeed.tech/articles/multilevel-graph-partitioning-on-gpu-supercomputers-142978.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24264>)

Author: Michael S. Gilbert, Erik G. Boman, Kamesh Madduri

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [graph](<https://devfeed.tech/tags/graph.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [supercomputers](<https://devfeed.tech/tags/supercomputers.md>)

### AI overview

The work presents a distributed multi-GPU graph partitioner based on the single-GPU Jet partitioner, aiming to improve scalability while preserving solution quality. It uses CUDA-aware MPI, one-sided communication, and a bulk-synchronous parallel structure. The authors report solution quality comparable to Jet and weak scaling up to 256 GPUs, allowing it to partition instances larger than a single-GPU partitioner can handle.

### Source excerpt

arXiv:2611.24264v1 Announce Type: new Abstract: Graph partitioning is a key enabler in computational science, with applications to sparse linear algebra, load-balanced parallel computation, and graph computations. In recent years, graphics processing units (GPUs) have become the preferred accelerator on the world's top supercomputers. Despite this, most existing distributed graph partitioners do not utilize GPUs. In this work, we develop a new multi-GPU distributed partitioner based on Jet, a prior single-GPU partitioner. We seek to optimize scalability without sacrificing quality. Our partitioner utilizes CUDA-aware MPI, one-sided communication routines, and an overall bulk-synchronous parallel structure. We demonstrate solution quality comparable to Jet. Our experiments show excellent weak scaling up to 256 GPUs, enabling partitioning of instances significantly larger than a single-GPU partitioner can handle.

## The Cost of Synchronization in GPU Scale-Up Domains

DevFeed: [The Cost of Synchronization in GPU Scale-Up Domains](<https://devfeed.tech/articles/the-cost-of-synchronization-in-gpu-scale-up-domains-142963.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.22503>)

Author: Arjun Devraj, Lindsey Bowen, Rachee Singh

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [synchronization](<https://devfeed.tech/topics/synchronization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gemm](<https://devfeed.tech/tags/gemm.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [scale-up](<https://devfeed.tech/tags/scale-up.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>), [tax](<https://devfeed.tech/tags/tax.md>)

### AI overview

Across 300,000 collectives from four language-model workloads on A100, H100, and H200 domains, GPUs reached the same collective hundreds to thousands of microseconds apart despite running identical kernels. Early arrivals wait for the slowest GPU, a synchronization tax that consumed nearly 40% of tensor-parallel communication time in an 8-GPU domain. An operation-chain analysis attributed 78% of this tax to cross-GPU variation in GEMM runtimes. The authors' model finds the tax grows at least logarithmically with domain size; for 72 GPUs, keeping communication below 12% of step time requires 3.3x the per-GPU bandwidth predicted by current simulators. They conclude scale-up fabrics should allocate switch capacity to fewer GPUs with more bandwidth each.

### Source excerpt

arXiv:2608.22503v2 Announce Type: replace Abstract: GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.

## GPU architecture, clearly explained

DevFeed: [GPU architecture, clearly explained](<https://devfeed.tech/articles/gpu-architecture-clearly-explained-141433.md>)

Original publisher: [Read original article](<https://blog.dailydoseofds.com/p/gpu-architecture-clearly-explained>)

Author: Avi Chawla

Published: 2026-11-19T00:13:29Z

Content type: article

Language: en

Sources: [Daily Dose of Data Science](<https://devfeed.tech/sources/daily-dose-of-data-science.md>)

Topics: [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [addition](<https://devfeed.tech/tags/addition.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arrays](<https://devfeed.tech/tags/arrays.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [block](<https://devfeed.tech/tags/block.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data](<https://devfeed.tech/tags/data.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-architecture](<https://devfeed.tech/tags/gpu-architecture.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-bandwidth](<https://devfeed.tech/tags/memory-bandwidth.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

For single-response LLM generation, memory bandwidth can limit token rate because each 16-bit weight read transfers two bytes for only two arithmetic operations. Batching requests improves reuse by applying each loaded weight across several requests. GPU execution is organized as threads grouped into 32-thread warps, blocks of cooperating warps, and SMs that run blocks. Registers, shared memory and L1, and the shared L2 cache keep data closer to compute; model weights and large arrays reside in HBM, which takes longer to access. Algorithms can reduce HBM traffic by reusing local data: FlashAttention processes smaller pieces and keeps intermediate results near compute instead of writing and rereading a full attention-score table. Thus compute ratings alone do not predict token speed; data movement and reuse also matter.

### Source excerpt

...and why memory bandwidth, not compute, sets your token rate.

## JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving

DevFeed: [JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving](<https://devfeed.tech/articles/jigslo-solving-the-resource-puzzle-for-efficient-multi-llm-serving-140374.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.06261>)

Author: Kyungmin Bin, Seungbeom Choi, Jimyoung Son, Jieun Choi, Daseul Bae, Daehyeon Baek, Kihyo Moon, Minsung Jang, Hyojung Lee

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decode](<https://devfeed.tech/tags/decode.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [resources](<https://devfeed.tech/tags/resources.md>)

### AI overview

JigSLO models how memory-bandwidth contention from co-located heterogeneous LLM instances affects latency, estimating it from contended and uncontended periods using standalone profiles. It schedules prefill and decode as separate instances, jointly choosing tensor parallelism, compute share, batch configuration, and co-location to meet latency targets while maximizing throughput. Its resource manager shares weights, pools KV cache, and transfers it between prefill and decode instances. Across chat, code-generation, and long-context summarization datasets, the paper reports average gains of 2.3x in SLO attainment and 1.2x in throughput over state-of-the-art baselines; on long-context workloads, it matches their best attainment with half the GPUs.

### Source excerpt

arXiv:2509.06261v3 Announce Type: replace Abstract: GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by $2.3\times$ in SLO attainment and $1.2\times$ in throughput on average, matching their best attainment with half the GPUs on long-context workloads.

## Supercharging Olmo-core for Efficient and Scalable MoE Training

DevFeed: [Supercharging Olmo-core for Efficient and Scalable MoE Training](<https://devfeed.tech/articles/supercharging-olmo-core-for-efficient-and-scalable-moe-training-136978.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18956>)

Author: Tianhua Tao, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas Samaga, Jacob Morrison, Iz Beltagy, Taira Anderson, Noah A. Smith, Hannaneh Hajishirzi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [b300](<https://devfeed.tech/tags/b300.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [ddp](<https://devfeed.tech/tags/ddp.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [fsdp](<https://devfeed.tech/tags/fsdp.md>), [gerrymandering](<https://devfeed.tech/tags/gerrymandering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvl8](<https://devfeed.tech/tags/nvl8.md>), [nvshmem](<https://devfeed.tech/tags/nvshmem.md>), [pipeline-parallelism](<https://devfeed.tech/tags/pipeline-parallelism.md>), [router](<https://devfeed.tech/tags/router.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Olmo-core's redesigned MoE training stack targets costs that grow with stored expert parameters even when per-token compute stays nearly constant. It replaces earlier FSDP with DDP plus expert and pipeline parallelism, a distributed optimizer, and activation recompute. NVSHMEM-based rowwise expert parallelism writes tokens directly to expert buffers; GPU-resident routing metadata and device-scheduled grouped GEMMs remove synchronization. Integrated MXFP8 reduces compute, activation-memory, and transport costs, while topology-agnostic checkpoints allow parallel layouts to change between runs. On NVL8 B300 nodes, configurations span 12.9 billion to 1.2 trillion parameters across up to 512 GPUs; the 1.2-trillion configuration reaches 858 useful-model TFLOP/s/GPU with MXFP8 and per-layer recompute. An experimental test with optional DeepEP v2 reaches 2.38 trillion parameters. The authors also report router exploitation of the load-balancing loss ("Token Gerrymandering") and warn that communication overlap can slow kernels more than it hides.

### Source excerpt

arXiv:2611.18956v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismatch is costly: in early baselines, throughput fell as stored expert parameter size grew, even though the useful work per token stayed nearly fixed. To close this gap, we present a redesigned training stack in Olmo-core, built for efficient, scalable MoE training. We replace the Fully Sharded Data Parallel (FSDP) of earlier Olmo-core versions with an architecture built on Distributed Data Parallel (DDP), and add Expert Parallelism (EP), Pipeline Parallelism (PP), a distributed optimizer, and activation recompute. We propose NVSHMEM-based rowwise EP, which writes each token directly into its expert's buffer, avoiding dispatch and combine permutation overhead. We make EP fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs. Closely integrated MXFP8 cuts compute, activation-memory, and transport costs, and topology-agnostic checkpoints let the parallel layout change between runs. On NVL8 B300 nodes, the stack supports training configurations from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, reaching 858 useful-model TFLOP/s/GPU at 1.2 trillion with MXFP8 and per-layer recompute; an experimental capacity test with the optional DeepEP v2 backend reaches 2.38 trillion. We also explain each design choice and share findings for other training systems; for example, we identify a failure pattern we call Token Gerrymandering, in which the MoE router can learn to hack the load-balancing loss, and find that overlapping communication with computation can slow the overlapped kernels by more than it hides.

## CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1

DevFeed: [CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1](<https://devfeed.tech/articles/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-training-v6-1-136727.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-r-training-v6-1>)

Author: CoreWeave

Published: 2026-11-16T19:52:27Z

Content type: news

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>), [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [NVIDIA Spectrum-X Ethernet](<https://devfeed.tech/topics/nvidia-spectrum-x-ethernet.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-provider](<https://devfeed.tech/tags/cloud-provider.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [first](<https://devfeed.tech/tags/first.md>), [full](<https://devfeed.tech/tags/full.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [leading](<https://devfeed.tech/tags/leading.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera](<https://devfeed.tech/tags/nvidia-vera.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [rack](<https://devfeed.tech/tags/rack.md>), [results](<https://devfeed.tech/tags/results.md>), [rubin](<https://devfeed.tech/tags/rubin.md>), [training](<https://devfeed.tech/tags/training.md>), [vera](<https://devfeed.tech/tags/vera.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>)

### AI overview

In MLPerf Training v6.1, CoreWeave reports training DeepSeek-V3 671B to target quality in 17.65 minutes on 256 Vera Rubin GPUs across four racks, versus 48.40 minutes on the same number of GB200 GPUs (2.7x faster, per CoreWeave's calculation). It also reports the round's fastest 72-GPU Llama 3.1 8B submission: 7.16 minutes on one Vera Rubin rack, compared with 13.70 minutes on GB200. The article attributes multi-rack performance to an integrated stack: Spectrum-X RoCE networking and ConnectX-9 SuperNICs, topology-aware placement and orchestration, optimized software, and fleet monitoring and recovery. On GB200, it reports Qwen3.5 397B reinforcement learning in 22.78 minutes across four racks, split evenly between training and generation using NeMo and TensorRT-LLM. These are CoreWeave's descriptions of published results; its speedup and fastest-submission claims are its own analyses and were not verified by MLCommons.

### Source excerpt

See how CoreWeave's full-stack optimizations delivered leading NVIDIA Vera Rubin performance in MLPerf® Training v6.1, from a single 72-GPU rack to multi-rack scale at 256 GPUs

## NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0

DevFeed: [NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0](<https://devfeed.tech/articles/nvidia-vera-rubin-nvl72-leads-agentic-inference-performance-on-mlperf-endpoints-v1-0-136678.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/nvidia-vera-rubin-nvl72-leads-agentic-inference-performance-on-mlperf-endpoints-v1-0/>)

Author: Zhihan Jiang

Published: 2026-11-16T16:00:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [agentic-tasks](<https://devfeed.tech/tags/agentic-tasks.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blackwell-ultra](<https://devfeed.tech/tags/blackwell-ultra.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context](<https://devfeed.tech/tags/context.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [dynamo](<https://devfeed.tech/tags/dynamo.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [groq-3-lpx](<https://devfeed.tech/tags/groq-3-lpx.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [infiniband](<https://devfeed.tech/tags/infiniband.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [tensorrt-llm](<https://devfeed.tech/tags/tensorrt-llm.md>), [tool](<https://devfeed.tech/tags/tool.md>), [top-stories](<https://devfeed.tech/tags/top-stories.md>), [turns](<https://devfeed.tech/tags/turns.md>), [vera-rubin-nvl72](<https://devfeed.tech/tags/vera-rubin-nvl72.md>)

### AI overview

Provisional MLPerf Endpoints v1.0 results report that NVIDIA Vera Rubin NVL72 delivered over 10x the throughput per kilowatt of GB300 NVL72 at a shared minimum average interactivity of 50 tokens per second per user. The comparison uses different configurations and concurrency: Vera Rubin's 32 accelerators served 1,024 queries at 50.4 tokens per second per user, while GB300's 48 accelerators served 48 queries at 56.2; the Vera Rubin results are pending MLCommons verification and peer review. The benchmark checks generated tool calls and workflow intent, with separate SWE-bench Verified evaluations for end-to-end task solving. The excerpt attributes performance to hardware and serving co-design, including HBM4, Tensor Cores, NVLink, prefix caching, state reuse, Dynamo routing, disaggregated prefill and decode, and model-specific kernels. It describes comparisons across throughput and responsiveness without interpolating between measured points.

### Source excerpt

Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool...

## Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication

DevFeed: [Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication](<https://devfeed.tech/articles/zero-sm-nccl-a-systematic-guide-to-designing-and-deploying-sm-free-gpu-communication-134894.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18075>)

Author: Zhenhao He, Siyuan Shen, Ke Wen, Prajwal Singhania, Shiqing Fan, Youngeun Kwon, Sangkug Lym, Vasudevan Rengasamy, Vishnuswaroop Ramesh, Yicheng Qiang, Craig Belusar, Jeff Hammond, Torsten Hoefler

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [NCCL](<https://devfeed.tech/topics/nccl.md>), [rdma](<https://devfeed.tech/topics/rdma.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [apis](<https://devfeed.tech/tags/apis.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [building](<https://devfeed.tech/tags/building.md>), [communication](<https://devfeed.tech/tags/communication.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [deep](<https://devfeed.tech/tags/deep.md>), [designing](<https://devfeed.tech/tags/designing.md>), [domain](<https://devfeed.tech/tags/domain.md>), [domains](<https://devfeed.tech/tags/domains.md>), [free](<https://devfeed.tech/tags/free.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [guide](<https://devfeed.tech/tags/guide.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [sm](<https://devfeed.tech/tags/sm.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

Zero-SM NCCL avoids SM kernels by synchronizing with CUDA stream memory operations, using copy engines for transfers within a scale-up domain, and driving network RDMA through a stream-ordered CPU proxy. It supports AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8, collectives achieved up to 1.36x NCCL's legacy AllGather bandwidth and reduced median overlapped GEMM slowdown from 33% to 4%, with higher per-operation latency. In Megatron-LM, FSDP improved by up to 3.5% and pipeline parallelism by up to 15% with one-sided puts. The gains depend on workload: the approach benefits large transfers in GPU-bound workloads and deep pipelines with large bubbles, but loses on small messages and host-bound workloads.

### Source excerpt

arXiv:2611.18075v1 Announce Type: new Abstract: GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.

## HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs

DevFeed: [HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs](<https://devfeed.tech/articles/hetccl-efficient-llm-training-on-heterogeneous-vendor-gpus-134880.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2601.22585>)

Author: Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [llm-training](<https://devfeed.tech/tags/llm-training.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [rdma](<https://devfeed.tech/tags/rdma.md>)

### AI overview

HetCCL enables collective communication across NVIDIA and AMD GPUs using RDMA, without driver modifications or changes to existing deep learning applications. It unifies vendor-specific backends and introduces two mechanisms that leverage NVIDIA NCCL and AMD RCCL. In the reported multi-vendor cluster evaluation, HetCCL matched NCCL and RCCL performance in homogeneous setups and scaled in heterogeneous environments. The supplied text does not describe the mechanisms or provide detailed evaluation results.

### Source excerpt

arXiv:2601.22585v2 Announce Type: replace Abstract: The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.

## TailGuard-GDS: Tail-Safe Control of GPU Storage Data Movement

DevFeed: [TailGuard-GDS: Tail-Safe Control of GPU Storage Data Movement](<https://devfeed.tech/articles/tailguard-gds-tail-safe-control-of-gpu-storage-data-movement-134889.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17824>)

Author: Jiaqi Liu, Fan Yang, Xiaohang Shi, Xianzhe Zhang, Tao Jiang, Guojun Yuan, Zhan Wang, Guangming Tan

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPUDirect](<https://devfeed.tech/topics/gpudirect.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [NVMe](<https://devfeed.tech/topics/nvme.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpudirect-storage](<https://devfeed.tech/tags/gpudirect-storage.md>), [latency](<https://devfeed.tech/tags/latency.md>), [nvme](<https://devfeed.tech/tags/nvme.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

TailGuard-GDS selects semantically equivalent GPU storage data-movement plans for each application phase, using measured GPU-visible tail latency and revisiting choices as conditions change. Its profiling varies backend, request shape, and placement; resource reuse keeps lookup and switching overhead at microsecond scale. The need for adaptation arises because the preferred path changes with operation, request size, concurrency, placement, and contention, and bandwidth and tail-latency winners can differ. In reported evaluations, it reduced FSDP step p95 by 18.4% and GPU-idle p95 by 25.3% versus the strongest fixed path. Across five eight-GPU ZeRO-3 workloads, step p95 fell 7.9-24.6% versus fixed host staging and 28.4-42.6% versus fixed GDS. Compared with tuned DeepSpeed AIO, step p95 fell 5.6% and estimated throughput rose 6.1%.

### Source excerpt

arXiv:2611.17824v1 Announce Type: new Abstract: AI runtimes move state between NVMe storage and GPU memory on critical paths. GPUDirect Storage (GDS) removes host staging, yet our eight-H100 measurements show it is not consistently tail-optimal for semantically identical GPU-buffer operations. The preferred path varies with operation, request size, concurrency, placement, and resource contention; bandwidth and tail-latency winners can disagree. Frameworks specify what moves and when, while GPU I/O stacks expose mechanisms; neither combines phase semantics with measured tail behavior to choose a delivery plan. We present TailGuard-GDS, a runtime that selects a semantically equivalent plan per application phase. It profiles plans varying backend, request shape, and placement, minimizes GPU-visible tail latency under phase constraints, and revisits decisions as conditions change. Resource reuse keeps policy lookup and switching at microsecond-scale overhead. Integrated with PyTorch FSDP and DeepSpeed ZeRO-3, TailGuard reduces FSDP step p95 by 18.4% and GPU-idle p95 by 25.3% versus the strongest fixed path. Across five eight-GPU ZeRO-3 workloads, step p95 falls 7.9-24.6% versus fixed host staging and 28.4-42.6% versus fixed GDS. Against tuned DeepSpeed AIO, TailGuard reduces step p95 by 5.6% and raises estimated throughput by 6.1%.

## OrbitWeaver: LLM Agent Planning with Simulation Feedback for Optically Interconnected Orbital GPU Clusters

DevFeed: [OrbitWeaver: LLM Agent Planning with Simulation Feedback for Optically Interconnected Orbital GPU Clusters](<https://devfeed.tech/articles/orbitweaver-llm-agent-planning-with-simulation-feedback-for-optically-interconnected-orbital-gpu-clusters-134891.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17850>)

Author: Pengcheng Luo, Bowen Zhang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [communication](<https://devfeed.tech/tags/communication.md>), [computing](<https://devfeed.tech/tags/computing.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [data-center](<https://devfeed.tech/tags/data-center.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-clusters](<https://devfeed.tech/tags/gpu-clusters.md>), [gpu-computing](<https://devfeed.tech/tags/gpu-computing.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [offline](<https://devfeed.tech/tags/offline.md>), [planning](<https://devfeed.tech/tags/planning.md>), [simulation](<https://devfeed.tech/tags/simulation.md>)

### AI overview

OrbitWeaver is an offline LLM agent planner for GPU clusters linked by inter-satellite optical connections. It jointly chooses rank placement and communication order, using communication structure, execution dependencies, and simulation feedback to coordinate shared-link access with computation progress. The approach aims to reduce contention and downstream dependency blocking. In simulations, it lowered aggregate mean job completion time by 6.2-33.5% compared with CASSINI, Crux, and Symphony. These results are limited to the reported simulation experiments.

### Source excerpt

arXiv:2611.17850v1 Announce Type: new Abstract: Space-based data-center designs increasingly consider GPU computing formations connected by inter-satellite optical links. With limited neighboring connections, jobs assigned disjoint GPUs can still contend on shared relay paths. Rank placement and communication priorities jointly affect this contention and the readiness of downstream computation. We present OrbitWeaver, an offline large language model (LLM) agent planner that jointly optimizes rank placement and communication order. Guided by communication structure, execution dependencies, and simulation feedback, the agent constructs and refines plans that coordinate shared-link access with computation progress. This coordination reduces contention and dependency blocking, shortening mean job completion time (JCT). Simulation experiments on optically interconnected orbital GPU clusters show that OrbitWeaver reduces aggregate mean JCT by 6.2-33.5% relative to the state-of-the-art baselines CASSINI, Crux, and Symphony. These results demonstrate the value of LLM-guided planning for efficient concurrent computing in orbital GPU clusters.

## SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs

DevFeed: [SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs](<https://devfeed.tech/articles/syclkittens-a-tile-programming-model-for-programmers-and-coding-agents-on-intel-gpus-134882.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.04277>)

Author: Yehong Jiang, Sheng Chen, Fangwen Fu, Yen-Kuang Chen, Xinmin Tian, Stuart H. Sul, Simran Arora

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [fabric](<https://devfeed.tech/tags/fabric.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [layouts](<https://devfeed.tech/tags/layouts.md>), [llama](<https://devfeed.tech/tags/llama.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

In controlled Intel GPU experiments, giving coding agents SyclKittens' hardware-aware tile model improved their GEMM performance over raw SYCL. With the same agent, task and feedback budget, Opus 4.8 reached 82.1% of tuned oneDNN performance using SyclKittens, compared with 57.5% using raw SYCL. The model encodes methods for matrix-engine layouts, L1 prefetching and communication across GPU stacks, while keeping workload-specific scheduling programmable for engineers and agents to refine. Their resulting kernel suite reached approximately 96% of oneDNN's geometric-mean performance across GEMM shapes and ran Llama-3.1-8B inference 1.59x faster than torch.compile on one GPU, and up to 2.91x faster than a matched multi-GPU decode path using Intel oneCCL.

### Source excerpt

arXiv:2610.04277v2 Announce Type: replace Abstract: New AI accelerators arrive before the kernels that make them fast, because peak kernel performance requires architecture-specific expertise in operand pipelines and data-movement techniques. Coding agents can now write, compile, and tune kernels on their own, so they could greatly accelerate kernel development and optimization. What agents produce depends on the interfaces they are given. These interfaces may expose a machine's raw capabilities or encode the known-good methods for exploiting its hardware features efficiently. We test the performance impact of these two interfaces on Intel GPUs through controlled experiments with five coding models on three kernels. With raw SYCL and execution feedback, Opus 4.8, the strongest of the three models tested with Claude Code, writes a single-GPU GEMM kernel reaching only 57.5% of Intel's tuned oneDNN library. We present SyclKittens, a hardware-aware tile programming model for Intel GPUs that encodes these known-good methods, so its operations place operands in matrix-engine layouts, prefetch through the L1 cache, and communicate over the fabric that links GPU stacks into a node. With SyclKittens under the same agent, task, and feedback budget, the agent reaches 82.1%, showing that encoded methods turn hardware capabilities into performance. Workload-specific policies stay programmable, and engineers and agents jointly refine these schedules in SyclKittens to build a kernel suite that reaches approximately 96% of oneDNN in geometric mean across GEMM shapes. The suite runs Llama-3.1-8B inference 1.59 times faster than torch.compile on one GPU and up to 2.91 times faster than a matched multi-GPU decode path on Intel's oneCCL. SyclKittens is open source and available at https://github.com/intel/SyclKittens.

## Compiling Specialized GPU-Initiated Networking

DevFeed: [Compiling Specialized GPU-Initiated Networking](<https://devfeed.tech/articles/compiling-specialized-gpu-initiated-networking-134896.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18145>)

Author: Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [device-code](<https://devfeed.tech/tags/device-code.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gin](<https://devfeed.tech/tags/gin.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [h100](<https://devfeed.tech/tags/h100.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [libraries](<https://devfeed.tech/tags/libraries.md>), [lifetimes](<https://devfeed.tech/tags/lifetimes.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [networking](<https://devfeed.tech/tags/networking.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvshmem](<https://devfeed.tech/tags/nvshmem.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>)

### AI overview

Obaleth integrates GPU-initiated communication into a kernel's intermediate representation so a compiler can co-optimize communication with computation and generate specialized transport code. It schedules completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. The abstract reports that on H100s it sends small messages up to 3.3x faster; on 256 GPUs its all-to-all runs 1.25x and 2.3x faster than NVSHMEM and NCCL GIN, respectively, with lower memory and register use. In vLLM, serving Qwen3-235B on 16 B200 GPUs, it answers requests up to 1.29x faster than DeepEP.

### Source excerpt

arXiv:2611.18145v1 Announce Type: new Abstract: Machine learning systems specialize computation to the workload and hardware, but communication remains in generic libraries whose device code and transport decisions are not visible to high-level compilers. We present Obaleth, a compiler that makes GPU-initiated communication part of a kernel's intermediate representation, co-optimizes it alongside computation, and lowers it to specialized transport code. Obaleth plans completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. On H100 GPUs, Obaleth sends small messages up to 3.3$\times$ faster, and on 256 GPUs runs all-to-all 1.25$\times$ and 2.3$\times$ faster than NVSHMEM and NCCL GIN, with much smaller memory and register use. In vLLM, it serves Qwen3-235B on 16 B200 GPUs and answers requests up to 1.29$\times$ faster than with DeepEP.

## Multi-GPU Tridiagonal Partition Algorithm

DevFeed: [Multi-GPU Tridiagonal Partition Algorithm](<https://devfeed.tech/articles/multi-gpu-tridiagonal-partition-algorithm-134895.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18111>)

Author: Milena Veneva

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [partition](<https://devfeed.tech/tags/partition.md>)

### AI overview

The paper presents CUDA implementations of two multi-GPU approaches for solving linear systems with tridiagonal coefficient matrices: a parallel partition method and a recursive version. It reports experiments using one and two GPU cards across different system sizes and numbers of recursive steps, but the supplied excerpt does not state the experimental results.

### Source excerpt

arXiv:2611.18111v1 Announce Type: new Abstract: This paper presents a multi-GPU version and a multi-GPU recursive version of the parallel partition method for systems of linear algebraic equations with tridiagonal coefficient matrices and their implementations in CUDA. Computational experiments on one and two GPU cards, for different system sizes and with different number of recursive steps are conducted. The results are analyzed and discussed.

## Dell Pro Precision 9 T4 Review: 86 Xeon Cores and Two RTX PRO 6000 GPUs for Models Too Big for One Card

DevFeed: [Dell Pro Precision 9 T4 Review: 86 Xeon Cores and Two RTX PRO 6000 GPUs for Models Too Big for One Card](<https://devfeed.tech/articles/dell-pro-precision-9-t4-review-86-xeon-cores-and-two-rtx-pro-6000-gpus-for-models-too-big-for-one-card-133422.md>)

Original publisher: [Read original article](<https://www.storagereview.com/review/dell-pro-precision-9-t4-review>)

Author: Kevin OBrien

Published: 2026-11-13T22:08:22Z

Content type: article

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [RTX PRO](<https://devfeed.tech/topics/rtx-pro.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Xeon 600](<https://devfeed.tech/topics/xeon-600.md>), [ddr5](<https://devfeed.tech/topics/ddr5.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-development](<https://devfeed.tech/tags/ai-development.md>), [ai-leaderboard](<https://devfeed.tech/tags/ai-leaderboard.md>), [ddr5](<https://devfeed.tech/tags/ddr5.md>), [dell](<https://devfeed.tech/tags/dell.md>), [ecc](<https://devfeed.tech/tags/ecc.md>), [end-user](<https://devfeed.tech/tags/end-user.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [local-ai](<https://devfeed.tech/tags/local-ai.md>), [models](<https://devfeed.tech/tags/models.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-rtx-pro-6000-blackwell](<https://devfeed.tech/tags/nvidia-rtx-pro-6000-blackwell.md>), [precision](<https://devfeed.tech/tags/precision.md>), [pro](<https://devfeed.tech/tags/pro.md>), [review](<https://devfeed.tech/tags/review.md>), [rtx](<https://devfeed.tech/tags/rtx.md>), [rtx-pro](<https://devfeed.tech/tags/rtx-pro.md>), [rtx-pro-6000](<https://devfeed.tech/tags/rtx-pro-6000.md>), [tower](<https://devfeed.tech/tags/tower.md>), [workstation](<https://devfeed.tech/tags/workstation.md>), [xeon](<https://devfeed.tech/tags/xeon.md>)

### AI overview

The Dell Pro Precision 9 T4 combines an 86-core Xeon 698X, 256GB of DDR5-6400 ECC memory, and two 96GB RTX PRO 6000 Blackwell Max-Q GPUs in a tower listed at about $66,600 as configured. The second GPU nearly doubled peak serving throughput across tested models, cut a LoRA epoch from 31.5 to 19.7 minutes, and let MiniMax M2.7--a 129.3GB quantized model--run at 83.7 tokens per second for one user, versus 30.3 with one GPU and the remainder in system memory. Eight-channel memory bandwidth helped the T4 outperform the 96-core Threadripper PRO 9995WX in tested OpenFOAM, GROMACS, and y-cruncher workloads. The 7875 was faster in core-bound compression, rendering, and compilation; the T4's lower per-core performance also showed in Valkey. The review highlights strong cooling and service access, while noting that a single-GPU T4 configuration performs much like other systems using the same 300W card.

### Source excerpt

The Dell Pro Precision 9 T4 is the single-socket tower at the top of Dell's Pro Precision line, and our review unit is configured for local AI at a scale most workstations can't reach. It pairs Intel's 86-core Xeon 698X with 256GB of DDR5-6400 ECC memory, two 300W NVIDIA RTX PRO 6000 Blackwell Max-Q cards The post Dell Pro Precision 9 T4 Review: 86 Xeon Cores and Two RTX PRO 6000 GPUs for Models Too Big for One Card appeared first on StorageReview.com.

## OpenFold3 on Amazon SageMaker JumpStart: Predict biomolecular structures

DevFeed: [OpenFold3 on Amazon SageMaker JumpStart: Predict biomolecular structures](<https://devfeed.tech/articles/openfold3-on-amazon-sagemaker-jumpstart-predict-biomolecular-structures-133424.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/openfold3-on-amazon-sagemaker-jumpstart-predict-biomolecular-structures/>)

Author: Malav Shastri

Published: 2026-11-13T21:53:40Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker JumpStart](<https://devfeed.tech/topics/amazon-sagemaker-jumpstart.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [aws-sns](<https://devfeed.tech/topics/aws-sns.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [account](<https://devfeed.tech/tags/account.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-sagemaker-jumpstart](<https://devfeed.tech/tags/amazon-sagemaker-jumpstart.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cost](<https://devfeed.tech/tags/cost.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [dna](<https://devfeed.tech/tags/dna.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [jumpstart](<https://devfeed.tech/tags/jumpstart.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [post](<https://devfeed.tech/tags/post.md>), [preview](<https://devfeed.tech/tags/preview.md>), [proteins](<https://devfeed.tech/tags/proteins.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [rna](<https://devfeed.tech/tags/rna.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

OpenFold3-preview on SageMaker JumpStart serves protein, DNA, RNA, ligand, and complex structure predictions through asynchronous endpoints, which queue jobs and store results or errors in S3. The article outlines deployment prerequisites, request formats, confidence files, and multi-GPU use: a prediction uses one GPU, while a four-GPU ml.g6.12xlarge can process four requests concurrently. Network isolation prevents calls to external MSA servers; users can supply precomputed alignments generated on infrastructure they control. In the article's carbonic anhydrase II example, an MSA recovered the known binding mode while sequence-only prediction misfolded, but the authors caution this well-studied complex is not a novel-target benchmark. Scale-to-zero can reduce idle instance charges, though restarting adds delay; S3 and other charges still apply. The model is described as a research preview, and the post advises meeting organizational security and compliance requirements before using proprietary or regulated data.

### Source excerpt

Deploy OpenFold3-preview from Amazon SageMaker JumpStart to predict 3D structures of proteins, DNA, RNA, and ligand complexes in your own AWS account. The post covers asynchronous inference, multi-GPU parallelism, bringing your own MSAs for accuracy and privacy, and scaling to zero to reduce cost.

## Run GPU-Powered Black Hole Interferometry with NVIDIA cuPhoton

DevFeed: [Run GPU-Powered Black Hole Interferometry with NVIDIA cuPhoton](<https://devfeed.tech/articles/run-gpu-powered-black-hole-interferometry-with-nvidia-cuphoton-133566.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/run-gpu-powered-black-hole-interferometry-with-nvidia-cuphoton/>)

Author: Akhilesh Mishra

Published: 2026-11-13T21:17:17Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [GPUDirect](<https://devfeed.tech/topics/gpudirect.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [Image](<https://devfeed.tech/topics/image.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>)

Tags: [black-hole](<https://devfeed.tech/tags/black-hole.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cuda-x](<https://devfeed.tech/tags/cuda-x.md>), [cuphoton](<https://devfeed.tech/tags/cuphoton.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [interferometry](<https://devfeed.tech/tags/interferometry.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [simulation-modeling-design](<https://devfeed.tech/tags/simulation-modeling-design.md>), [supercomputing](<https://devfeed.tech/tags/supercomputing.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>)

### AI overview

NVIDIA cuPhoton.xBeam correlates radio recordings from telescope pairs into complex visibilities, which feed later calibration and image reconstruction. Its workflow indexes VDIF frames with reusable `.vdix` sidecars, applies job-specified delay and fringe corrections, channelizes voltages with FFTs, cross-multiplies and accumulates station pairs, then writes SWIN or `xbeam-v1` output. GPU processing uses the CuPy backend; data can be read through POSIX and pinned host memory or cuFile/GPUDirect Storage when supported by runtime conditions. xBeam can split scans across GPUs on one node or across nodes, then merge shards after checking job identity, time coverage, and headers. On a Vera Rubin NVL72 system, a 10 TB dataset was processed in 70 seconds on 64 GPUs, about 460 times faster than the stated x86 CPU baseline; this benchmark covers ingestion, preparation, and correlation. xBeam uses delays supplied in the job and does not calculate them from station positions or model black-hole light paths. Missing samples are not recovered; reader policy affects timing, visibilities, and reported weights. Calibration and imaging are outside the tutorial's scope.

### Source excerpt

Imaging black holes is challenging because they are distant and appear tiny in the sky--for example, M87*'s observed ring spans just 42 microarcseconds. The...

## When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

DevFeed: [When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM](<https://devfeed.tech/articles/when-one-gpu-isn-t-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm-132447.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/>)

Author: Fatima Fall

Published: 2026-11-13T17:00:00Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [DGX Station](<https://devfeed.tech/topics/dgx-station.md>), [gpudirect rdma](<https://devfeed.tech/topics/gpudirect-rdma.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [dgx](<https://devfeed.tech/tags/dgx.md>), [dgx-station](<https://devfeed.tech/tags/dgx-station.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [llm-benchmarking](<https://devfeed.tech/tags/llm-benchmarking.md>), [llms](<https://devfeed.tech/tags/llms.md>), [local-ai](<https://devfeed.tech/tags/local-ai.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [serving](<https://devfeed.tech/tags/serving.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The tutorial runs the 753-billion-parameter GLM-5.3 NVFP4 model across two GB300 DGX Stations because its roughly 464 GB checkpoint exceeds one GPU's 252 GB of memory. vLLM tensor parallelism splits weights and computation across both GPUs; ConnectX-8 RDMA carries rank-to-rank traffic, and Station A exposes a single OpenAI-compatible API. The workflow covers validating the link, downloading and syncing identical checkpoint copies, launching one vLLM rank per station, checking RDMA and API health, and measuring load with AIPerf on 80 SPEED-Bench coding conversations. Prerequisites include substantial disk space on each station, SSH access, Hugging Face access to the gated dataset, and acceptance of the model and dataset terms. Reported throughput peaks at 607.8 tokens/s at concurrency 16, then drops to 385.0 tokens/s at concurrency 32, where mean latency is 453.5 seconds; mean per-user throughput declines as concurrency rises. The excerpt notes that longer prompts or more concurrent users need more KV cache, and suggests reducing context length or request concurrency if memory runs out.

### Source excerpt

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...

## MegaOps: A Kernel Overlapping Framework with PyTorch-like Modularity and Megakernel Performance

DevFeed: [MegaOps: A Kernel Overlapping Framework with PyTorch-like Modularity and Megakernel Performance](<https://devfeed.tech/articles/megaops-a-kernel-overlapping-framework-with-pytorch-like-modularity-and-megakernel-performance-131544.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16248>)

Author: Jinkun Lin, Byungsoo Oh, Rachee Singh

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Code Modularity](<https://devfeed.tech/topics/code-modularity.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [best](<https://devfeed.tech/tags/best.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [cause](<https://devfeed.tech/tags/cause.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compose](<https://devfeed.tech/tags/compose.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cublas](<https://devfeed.tech/tags/cublas.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [framework](<https://devfeed.tech/tags/framework.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [like](<https://devfeed.tech/tags/like.md>), [modularity](<https://devfeed.tech/tags/modularity.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

MegaOps aims to retain kernel-per-operator modularity while removing whole-kernel dependency barriers. It launches dependent kernels on separate CUDA streams and synchronizes on individual intermediate-tensor tiles, allowing kernels with matching data tilings to be composed without modifying them. On dense Llama3 and Qwen3 across single H100, A100, and L40 GPUs and multi-GPU servers, the authors report throughput matching hand-tuned ThunderKittens megakernels; against the best-performing baseline, decode throughput is 1.09-1.10x and prefill throughput 0.98-1.10x. The framework reuses existing Triton, cuBLAS, and cuDNN kernels.

### Source excerpt

arXiv:2611.16248v1 Announce Type: new Abstract: Modern ML frameworks execute models using the kernel-per-operator paradigm, which runs each operator as a separate GPU kernel. This paradigm is modular, since developers can compose various kernel implementations and profile each kernel in isolation. However, this paradigm enforces dependencies at the granularity of entire kernels, which places a barrier between dependent kernels. These barriers cause streaming multiprocessors to idle, serialize computation and communication, and prevent a kernel from prefetching its weights while the preceding kernel runs. Megakernels remove kernel barriers by fusing the operators of an entire layer or model into a single kernel, but they give up the modularity of the kernel-per-operator paradigm. In this work, we ask whether a kernel-per-operator ML framework can perform on par with megakernels. Our key insight is that removing kernel barriers does not require fusing kernels. We develop MegaOps, a framework that launches dependent kernels on separate CUDA streams and enforces each dependency on individual tiles of the intermediate tensor that the producer kernel writes and the consumer kernel reads. Since kernels synchronize on data tiles instead of on each other's thread blocks, any two kernels that adopt the same data tiling can be composed without modifying either kernel. We evaluate MegaOps on dense Llama3 and Qwen3 models on single H100, A100, and L40 GPUs and on multi-GPU servers with 4$\times$A100 and 8$\times$H100. Our MegaOps framework matches the throughput of hand-tuned ThunderKittens megakernels, and achieves 1.09$\times$-1.10$\times$ the decode throughput and 0.98$\times$-1.10$\times$ the prefill throughput of the best-performing baseline, while reusing existing Triton, cuBLAS, and cuDNN kernels.

## SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference

DevFeed: [SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference](<https://devfeed.tech/articles/subgpu-pipelining-intra-gpu-fine-grained-spatial-parallelism-for-llm-inference-131525.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14904>)

Author: Muralidhar Andoorveedu, Marius Juston, Tingkai Liu, Sanjay Patel

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [changes](<https://devfeed.tech/tags/changes.md>), [chip](<https://devfeed.tech/tags/chip.md>), [choice](<https://devfeed.tech/tags/choice.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [fine](<https://devfeed.tech/tags/fine.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [pipelining](<https://devfeed.tech/tags/pipelining.md>), [spatial](<https://devfeed.tech/tags/spatial.md>)

### AI overview

SubGPU Pipelining divides an LLM operator graph into ordered segments assigned to stages with disjoint SM pools, then streams microbatches through them. Graph cuts, stage bindings, SM allocations, and microbatch size must be tuned together because they affect overlap, resource contention, cache state, and execution time. For GPT-OSS 120B under modeled 32-GPU B200 settings, its throughput frontier reaches up to 2.2x higher per-user throughput at matched per-GPU throughput, or 1.9x higher per-GPU throughput at matched per-user throughput, than temporal execution. A silicon experiment used one B200 to emulate the deployment and tested one two-stage configuration with CUDA Green Contexts and TensorRT-LLM kernels. At global batch 2,048, mean DRAM read-plus-write throughput rose from 71.8% to 83.6% of peak, and 36-layer decoder iteration time fell by 10.33%. The silicon result covers that selected configuration; the larger frontier gains are modeled.

### Source excerpt

arXiv:2611.14904v1 Announce Type: new Abstract: Large language model (LLM) decode is dominated by short, predominantly memory-bound kernels that make bursty use of on-chip execution resources and memory bandwidth. This paper introduces SubGPU Pipelining, a fine-grained, intra-GPU spatial execution model that cuts an operator graph into ordered contiguous segments, binds those segments to spatial stages backed by disjoint streaming-multiprocessor (SM) pools, and streams microbatches through the resulting schedule. Selecting the graph cuts, segment-to-stage binding, per-stage SM allocation, and microbatch size is a coupled optimization problem: each choice changes the overlap pattern, shared-resource contention, cache state, and execution times of the others. We evaluate SubGPU Pipelining through both simulation and silicon experiment using the GPT-OSS 120B workload with 32-GPU B200 deployment settings. The analytical shared-resource simulator searches across SubGPU pipeline structures and batch sizes to map the per-user/per-GPU throughput Pareto frontier. At matched per-GPU throughput, the modeled SubGPU frontier provides up to 2.2x higher per-user throughput than temporal execution, and at matched per-user throughput it provides up to 1.9x higher per-GPU throughput. To test physical feasibility and further validate the simulation results, silicon experiments are performed, using one B200 GPU to emulate a 32-GPU B200 deployment. The experiment evaluates one selected two-stage pipeline configuration at global batch 2,048 using CUDA Green Contexts and TensorRT-LLM kernels. Results show that SubGPU Pipelining improves mean DRAM read-plus-write throughput from 71.8% to 83.6% of peak, and reduces the 36-layer decoder iteration time by 10.33%.

## Snoopie: A Multi-GPU Communication Profiler and Visualizer

DevFeed: [Snoopie: A Multi-GPU Communication Profiler and Visualizer](<https://devfeed.tech/articles/snoopie-a-multi-gpu-communication-profiler-and-visualizer-131548.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16330>)

Author: Mohammad Kefah Taha Issa, Muhammad Aditya Sasongko, Ilyas Turimbetov, Javid Baydamirli, Do\\u{g}an Sa\\u{g}bili, Didem Unat

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [profiler](<https://devfeed.tech/tags/profiler.md>)

### AI overview

Snoopie fills gaps in multi-GPU profiling by tracking peer-to-peer transfers and GPU-centric communication-library calls that existing tools, despite their strength in profiling single-GPU compute, inadequately monitor. Built with NVBit instrumentation, it can attribute data movement to source-code lines and data objects, with visualizations ranging from system-wide summaries to specific instructions and addresses. Case studies show its use in monitoring transfers, finding performance bugs, and examining data movement hidden behind communication libraries.

### Source excerpt

arXiv:2611.16330v1 Announce Type: new Abstract: With data movement becoming one of the most expensive bottlenecks in computing, the need for profiling tools to analyze communication becomes crucial for effectively scaling multi-GPU applications. While existing profiling tools including first-party software by GPU vendors are robust and excel at capturing compute operations within a single GPU, support for monitoring GPU-GPU data transfers and calls issued by communication libraries is currently inadequate. To fill these gaps, we introduce Snoopie, an instrumentation-based multi-GPU communication profiling tool built on NVBit, capable of tracking peer-to-peer transfers and GPU-centric communication library calls. To increase programmer productivity, Snoopie can attribute data movement to the source code line and the data objects involved. It comes with multiple visualization modes at varying granularities, from a coarse view of the data movement in the system as a whole to specific instructions and addresses. Our case studies demonstrate Snoopie's effectiveness in monitoring data movement, locating performance bugs in applications, and understanding concrete data transfers abstracted beneath communication libraries. The tool is publicly available at https://github.com/ParCoreLab/snoopie.

## Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning

DevFeed: [Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning](<https://devfeed.tech/articles/starscream-reimagining-llm-sharding-on-amd-instinct-gpus-with-gpu-partitioning-131539.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15951>)

Author: Sumanth Gudaparthi, Smeet Chheda, Ganesh Dasika

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sharding](<https://devfeed.tech/topics/sharding.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NUMA](<https://devfeed.tech/topics/numa.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [attention](<https://devfeed.tech/tags/attention.md>), [chiplet](<https://devfeed.tech/tags/chiplet.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-partitioning](<https://devfeed.tech/tags/gpu-partitioning.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [llm](<https://devfeed.tech/tags/llm.md>), [locality](<https://devfeed.tech/tags/locality.md>), [memory](<https://devfeed.tech/tags/memory.md>), [numa](<https://devfeed.tech/tags/numa.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [performance](<https://devfeed.tech/tags/performance.md>), [redundancy](<https://devfeed.tech/tags/redundancy.md>), [sharding](<https://devfeed.tech/tags/sharding.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [tensor-parallelism](<https://devfeed.tech/tags/tensor-parallelism.md>)

### AI overview

Starscream adapts LLM sharding to partitioned, chiplet-based GPUs by adding intra-GPU sharding along an orthogonal dimension instead of extending inter-GPU sharding across the partitions. It targets three problems in conventional approaches: redundant memory use, reduced arithmetic intensity, and communication overhead from flat collectives. Its design principles are to preserve kernel shapes, eliminate memory redundancy, and minimize communication. In a tensor-parallelism case study on AMD MI300X and MI355X, partition-aware designs reportedly achieved up to 4x speedup for attention and 1.43x for fused GEMMs versus chiplet-oblivious baselines. The supplied abstract claims the principles can extend to other sharding strategies and future architectures; it does not provide broader evaluation details.

### Source excerpt

arXiv:2611.15951v1 Announce Type: new Abstract: As GPU accelerators grow larger to meet rising compute demands, they exhibit increasingly pronounced non-uniform memory access (NUMA) characteristics that challenge existing LLM frameworks. Chiplet-based architectures provide a practical path to larger GPUs while mitigating manufacturing yield constraints. Modern GPUs such as AMD Instinct MI300X employs such a multi-die design, and further offer GPU partitioning capabilities, partitioning a single physical GPU into multiple logical devices with NUMA-aware memory domains, promising faster kernel launches, and reduced memory access latencies with improved locality. However, current parallelization strategies, designed for homogeneous GPU topologies, when applied under partitioning, extend its unidimensional sharding causing significant performance degradation. Three bottlenecks emerge: (1) memory redundancy; (2) arithmetic intensity collapse; (3) communication overhead from flat collectives. We present Starscream, a rethinking of LLM sharding for the chiplet era. Rather than stretching inter-GPU sharding thinner, Starscream adds intra-GPU sharding along an orthogonal dimension, grounded in three foundational principles: kernel shape preservation, memory redundancy elimination, and communication overhead minimization. These principles form the basis for extending any sharding strategy into the partitioning regime. We demonstrate their effectiveness using tensor parallelism as a case study. On AMD MI300X and MI355X, exploiting partitioning capabilities yields up to 4x speedup for attention and 1.43x for fused GEMMs compared to chiplet-oblivious baselines, demonstrating the effectiveness of chiplet-aware designs. Starscream's principles generalize across sharding strategies and future architectures.

## Rethinking Memory For AI Inference

DevFeed: [Rethinking Memory For AI Inference](<https://devfeed.tech/articles/rethinking-memory-for-ai-inference-130168.md>)

Original publisher: [Read original article](<https://semiengineering.com/rethinking-memory-for-ai-inference/>)

Author: Liz Allan

Published: 2026-11-12T08:11:46Z

Content type: article

Language: en

Sources: [Semiconductor Engineering](<https://devfeed.tech/sources/semiconductor-engineering.md>)

Topics: [Local AI](<https://devfeed.tech/topics/local-ai.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [CPU Cache](<https://devfeed.tech/topics/cpu-cache.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [about](<https://devfeed.tech/tags/about.md>), [abstraction](<https://devfeed.tech/tags/abstraction.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [arteris](<https://devfeed.tech/tags/arteris.md>), [ayar-labs](<https://devfeed.tech/tags/ayar-labs.md>), [baya-systems](<https://devfeed.tech/tags/baya-systems.md>), [boost](<https://devfeed.tech/tags/boost.md>), [but](<https://devfeed.tech/tags/but.md>), [cadence](<https://devfeed.tech/tags/cadence.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [compute-in-memory](<https://devfeed.tech/tags/compute-in-memory.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [expedera](<https://devfeed.tech/tags/expedera.md>), [in-memory-compute](<https://devfeed.tech/tags/in-memory-compute.md>), [inference](<https://devfeed.tech/tags/inference.md>), [keysight-eda](<https://devfeed.tech/tags/keysight-eda.md>), [low-power-high-performance](<https://devfeed.tech/tags/low-power-high-performance.md>), [lpddr](<https://devfeed.tech/tags/lpddr.md>), [memory](<https://devfeed.tech/tags/memory.md>), [near-memory](<https://devfeed.tech/tags/near-memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [power](<https://devfeed.tech/tags/power.md>), [processing](<https://devfeed.tech/tags/processing.md>), [processing-in-memory](<https://devfeed.tech/tags/processing-in-memory.md>), [processing-near-memory](<https://devfeed.tech/tags/processing-near-memory.md>), [quadric](<https://devfeed.tech/tags/quadric.md>), [rambus](<https://devfeed.tech/tags/rambus.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [semiconductor](<https://devfeed.tech/tags/semiconductor.md>), [semiconductor-engineering](<https://devfeed.tech/tags/semiconductor-engineering.md>), [sk-hynix](<https://devfeed.tech/tags/sk-hynix.md>), [software](<https://devfeed.tech/tags/software.md>), [thermal](<https://devfeed.tech/tags/thermal.md>), [top-stories](<https://devfeed.tech/tags/top-stories.md>), [vinci](<https://devfeed.tech/tags/vinci.md>)

### AI overview

Moving AI inference compute closer to memory can reduce data movement, latency, and energy use, which matters for edge devices with limited power. The best fit depends on the workload: PIM and near-memory approaches can help low-batch decoding, while large-batch decode and prefill may become activation-communication bottlenecks. LPDDR6-PIM targets low-power applications with simple MAC operations; custom HBM logic base dies allow broader functions but face thermal limits. Adoption also requires compilers, operating systems, memory controllers, and system protocols to support offloading and scheduling. Hardware and software complexity, manufacturing cost, workload matching, and heat remain constraints, and broad commercial use is still early.

### Source excerpt

Processing data closer to memory could reduce power and boost performance, but software, thermal, and architectural challenges remain. The post Rethinking Memory For AI Inference appeared first on Semiconductor Engineering.

[Next page](<https://devfeed.tech/topics/multi-gpu.md?cursor=WyIyMDI2LTExLTEyVDA4OjExOjQ2KzAwOjAwIiwgIjU4MWRhYzUxLWFiOTQtNDkzOS1iZGVhLTM2NjJkNjg2ZWRhNyJd>)