# distributed-training

Distributed training is a machine-learning technique that splits model-training workloads across multiple worker nodes, GPUs, machines, or TPUs operating in parallel.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?

DevFeed: [RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?](<https://devfeed.tech/articles/rsibench-infra-can-agents-optimize-their-own-training-and-inference-infrastructure-at-production-scale-143519.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24395>)

Author: Hanlin Wu, Yifei Wang, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Hao Zhou

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infra](<https://devfeed.tech/tags/infra.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RSIBench-Infra evaluates whether agents can improve end-to-end training and inference performance under production workloads. It covers 20 tasks across model families and modalities, with deployments of up to 32 NVIDIA GPUs or Ascend NPUs. Agents begin from runnable baselines based on official recipes and may change the engine, kernel, or instruction levels; OCI image-layer submissions enable reproducible deployment, and independent evaluation checks end-to-end performance against task-specific quality constraints. Across task results, 66.2% achieved validated speedups above 1.10x. The abstract reports that agents can improve production-scale performance, while finding effective strategies and turning local gains into consistent end-to-end improvements remain challenging.

### Source excerpt

arXiv:2611.24395v1 Announce Type: new Abstract: Improving the efficiency of training and inference infrastructure is a practical component of recursive self-improvement. Existing benchmarks for agent-driven optimization largely focus on individual software layers such as kernel, leaving unclear whether agents can deliver end-to-end performance gains under production workloads. We introduce RSIBench-Infra, a benchmark of 20 training and inference tasks spanning diverse model families and modalities, with deployments of up to 32 accelerators on NVIDIA GPUs and Ascend NPUs. Agents start from runnable baselines derived from official recipes and can modify the software stack across engine, kernel, and instruction levels. An OCI image-layer submission protocol captures these changes for reproducible deployment, while independent evaluation measures end-to-end performance under task-specific quality constraints. Across all task results, 66.2% exceed $1.10\times$ validated speedup. These results demonstrate that current agents can improve production-scale training and inference performance, but identifying effective optimization strategies and consistently translating local improvements into end-to-end gains remain challenging.

## The Cost of Synchronization in GPU Scale-Up Domains

DevFeed: [The Cost of Synchronization in GPU Scale-Up Domains](<https://devfeed.tech/articles/the-cost-of-synchronization-in-gpu-scale-up-domains-142963.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.22503>)

Author: Arjun Devraj, Lindsey Bowen, Rachee Singh

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [synchronization](<https://devfeed.tech/topics/synchronization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gemm](<https://devfeed.tech/tags/gemm.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [scale-up](<https://devfeed.tech/tags/scale-up.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>), [tax](<https://devfeed.tech/tags/tax.md>)

### AI overview

Across 300,000 collectives from four language-model workloads on A100, H100, and H200 domains, GPUs reached the same collective hundreds to thousands of microseconds apart despite running identical kernels. Early arrivals wait for the slowest GPU, a synchronization tax that consumed nearly 40% of tensor-parallel communication time in an 8-GPU domain. An operation-chain analysis attributed 78% of this tax to cross-GPU variation in GEMM runtimes. The authors' model finds the tax grows at least logarithmically with domain size; for 72 GPUs, keeping communication below 12% of step time requires 3.3x the per-GPU bandwidth predicted by current simulators. They conclude scale-up fabrics should allocate switch capacity to fewer GPUs with more bandwidth each.

### Source excerpt

arXiv:2608.22503v2 Announce Type: replace Abstract: GPU scale-up domains are the building blocks of machine learning infrastructure. Architects provision the size and bandwidth in these domains assuming all GPUs reach a collective operation at the same time. However, we find that this assumption does not hold in practice. Across 300,000 collectives from training four language models on A100, H100, and H200 domains, GPUs reach the same collective hundreds to thousands of microseconds apart, despite running identical kernels. Since a collective cannot complete until every GPU contributes its data, GPUs that arrive early wait for the slowest one. We call this waiting time the synchronization tax, and it consumes nearly 40% of tensor-parallel communication time in an 8-GPU domain. To find its source, we develop an algorithm that recovers the chain of operations that delays each GPU before a collective. Our algorithm attributes 78% of the tax to variation in the runtimes of matrix-multiplication (GEMM) kernels across GPUs. Since the tax depends on the slowest of many GPUs, our analytical model shows that it grows at least logarithmically with domain size. Thus, a 72-GPU domain requires 3.3x more per-GPU bandwidth than today's simulators predict to keep communication below 12% of step time, and the optimal scale-up fabric should partition a switch's capacity among fewer GPUs with more bandwidth each.

## Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

DevFeed: [Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs](<https://devfeed.tech/articles/systolic-linear-attention-co-designing-tile-grouped-gated-delta-attention-for-tpus-143344.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06828>)

Author: Gagik Amirkhanyan, Nuojin Cheng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [delta](<https://devfeed.tech/tags/delta.md>), [deltanet](<https://devfeed.tech/tags/deltanet.md>), [gated-deltanet](<https://devfeed.tech/tags/gated-deltanet.md>), [google](<https://devfeed.tech/tags/google.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hbm](<https://devfeed.tech/tags/hbm.md>), [high-bandwidth-memory](<https://devfeed.tech/tags/high-bandwidth-memory.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-attention](<https://devfeed.tech/tags/linear-attention.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [tensor-core](<https://devfeed.tech/tags/tensor-core.md>), [tpus](<https://devfeed.tech/tags/tpus.md>)

### AI overview

Systolic Linear Attention (SLA) reshapes gated-delta linear attention for Google TPUs' wide 256x256 matrix units. It widens keys to 256, shares keys and scalar write gates across grouped value heads to reduce repeated chunk solves, groups decay gates into four 64-channel tiles, and fuses exact FP32 chunk inversion with backward reuse in one on-chip kernel. On TPU v7x, two-stage SLA-R4 was 2.1-2.6x faster than open-source MaxText KDA and 1.4-2.4x faster than Tokamax KDA, using 49-61% less temporary HBM. In a 2,048-chip pre-training study of sparse MoE models from 6.5B to 63.1B nominal parameters, it trained 11-24% faster than MaxText KDA and 4-21% faster than Tokamax KDA. Perplexity matched KDA across three paired seeds at 12.5B and was within 3% at other tested scales.

### Source excerpt

arXiv:2611.06828v2 Announce Type: replace Abstract: Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make unpacked heads, per-channel decay rescaling, and per-value-head chunk solves cheap. On Google TPUs, which concentrate compute in wide $256 \times 256$ systolic matrix units backed by a slower vector unit, direct ports leave half of each matrix tile padded (for $d_k = 128$ heads) and repeat decay rescaling and chunk solves across every value head. We present \textbf{Systolic Linear Attention (SLA)}, which reshapes gated delta attention around the matrix unit by widening keys to $256$ and sharing both keys and scalar write gates $\beta_t$ across grouped value heads ($R\times$ fewer chunk solves), grouping decay gates into four $64$-channel tiles, and fusing exact FP32 block-doubling chunk inversion and its backward reuse inside one on-chip kernel. With exact chunk inversion on TPU v7x, $2$-stage \texttt{SLA-R4} (four value heads per key head) runs $2.1\times$--$2.6\times$ faster than open-source MaxText KDA and $1.4\times$--$2.4\times$ faster than open-source Tokamax KDA, while using $49\%$--$61\%$ less temporary high-bandwidth memory (HBM). Across a $2{,}048$-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models from $6.5\text{B}$ to $63.1\text{B}$ nominal parameters, SLA trains $11\%$--$24\%$ faster than MaxText KDA ($4\%$--$21\%$ faster than Tokamax KDA), matches KDA perplexity on three paired seeds at $12.5\text{B}$, and stays within $3\%$ at other scales.

## DexAttention: Distributed Sparse Attention through Rapid Online Planning and Persistent GPU Execution

DevFeed: [DexAttention: Distributed Sparse Attention through Rapid Online Planning and Persistent GPU Execution](<https://devfeed.tech/articles/dexattention-distributed-sparse-attention-through-rapid-online-planning-and-persistent-gpu-execution-142968.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23512>)

Author: Sirui Chen, Xuehai Qian

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [execution](<https://devfeed.tech/tags/execution.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [online](<https://devfeed.tech/tags/online.md>), [planning](<https://devfeed.tech/tags/planning.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>)

### AI overview

DexAttention co-designs online planning and persistent GPU execution to reduce communication and per-operation overhead in distributed sparse attention. Its GPU accelerated, topology aware assigner balances compute and communication; a greedy scheduler orders coarse operations, while persistent kernels enforce fine grained dependencies and dispatch ready tasks. This separates detailed synchronization from planning and supports overlap without repeated attention launches. The framework supports static and dynamic per head masks across multiple attention kernels. On a four node, 16 GPU cluster, it reports a 1.39x geometric mean end to end training speedup over the fastest applicable baseline across 11 configurations. For dynamic video attention, it reports 2.11-2.37x speedups over Ring Attention, including planning.

### Source excerpt

arXiv:2611.23512v1 Announce Type: new Abstract: Sparse attention reduces computation, but distributed execution may still transfer unnecessary data and incur substantial per-operation overhead. Mask-aware assignment and scheduling can reduce these costs, but replanning for dynamic masks must be cheaper than the execution time it saves. We present DexAttention, a distributed sparse attention system that co-designs fast online planning with persistent execution. Its assigner uses GPU-accelerated, topology-aware clustering to balance computational load and reduce communication. A lightweight greedy scheduler orders coarse communication and computation operations, while a persistent kernel on each GPU enforces fine-grained dependencies and dispatches ready tasks. This division keeps detailed synchronization out of planning while enabling overlap without repeated attention launches. The framework supports static and dynamic per-head masks across multiple attention compute kernels. On a four-node, 16-GPU cluster, DexAttention achieves a $1.39\times$ geometric-mean end-to-end training speedup over the fastest applicable baseline across 11 configurations. For dynamic video attention, it achieves $2.11$--$2.37\times$ speedups over Ring Attention including planning.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.

## DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

DevFeed: [DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency](<https://devfeed.tech/articles/daisyrose-27b-pushing-the-frontier-of-post-training-efficiency-143008.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/RadixArk/daisyrose-27b-pushing-the-frontier-of-post-trainin>)

Author: Jason Chen

Published: 2026-11-18T19:54:56Z

Content type: release

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-tasks](<https://devfeed.tech/tags/agent-tasks.md>), [apache-2-0](<https://devfeed.tech/tags/apache-2-0.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [backend](<https://devfeed.tech/tags/backend.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [qwen3-8](<https://devfeed.tech/tags/qwen3-8.md>), [qwen3-8-27b](<https://devfeed.tech/tags/qwen3-8-27b.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [terminal-bench](<https://devfeed.tech/tags/terminal-bench.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RadixArk reports that DaisyRose-27B, built on Qwen3.8-27B and trained with Miles, raised Terminal-Bench 4.1 pass@1 from 5.6% to 12.0%. Training used 1,200 curated long-horizon agent tasks over 18 days on 144 H200 GPUs; the tasks targeted original-model errors, required multi-step work, and were checked to ensure successful completion was necessary. The model retains the base architecture, tokenizer, chat template, image understanding, and 256K context window; checkpoints are Apache 2.0. RadixArk says its internal backend achieved 37% MFU versus 16% for the open-source Megatron-based trainer on comparable runs, and roughly twice the end-to-end throughput. Its sandbox service reported 99% creation success over nine weeks, exceeding 99.95% since late October, and about 80% lower infrastructure costs than popular cloud sandbox providers. The article recommends SGLang with DSpark speculative decoding and points to the model card for inference settings and evaluation details.

### Source excerpt

Today we are releasing **DaisyRose-27B**, RadixArk's first model, built on Qwen3.8-27B and trained with reinforcement learning using Miles. We trained it on long-horizon agent tasks with sessions up to 1M tokens, with just 1,200 curated tasks in 18 days on 144 H200 GPUs. DaisyRose-27B improves Terminal-Bench 4.1 pass@1 from **5.6% to 12.0%**. Beyond Terminal-Bench 4.1, it also delivers gains on a variety of other reasoning, instruction following, and coding benchmarks.

## SPICE: Sparse Parameter Gossip for LLM Pretraining

DevFeed: [SPICE: Sparse Parameter Gossip for LLM Pretraining](<https://devfeed.tech/articles/spice-sparse-parameter-gossip-for-llm-pretraining-140393.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23053>)

Author: Pietro Cagnasso, Daniele Kopyshevskiy, Edouard Oyallon, Eugene Belilovsky

Published: 2026-11-18T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [bandwidth-constraints](<https://devfeed.tech/tags/bandwidth-constraints.md>), [better](<https://devfeed.tech/tags/better.md>), [combines](<https://devfeed.tech/tags/combines.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [ddp](<https://devfeed.tech/tags/ddp.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [error](<https://devfeed.tech/tags/error.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [full](<https://devfeed.tech/tags/full.md>), [global](<https://devfeed.tech/tags/global.md>), [internet](<https://devfeed.tech/tags/internet.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [less](<https://devfeed.tech/tags/less.md>), [link](<https://devfeed.tech/tags/link.md>), [llm](<https://devfeed.tech/tags/llm.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [spice](<https://devfeed.tech/tags/spice.md>)

### AI overview

SPICE reduces communication costs in decentralized LLM pretraining by exchanging sparse parameter updates and using error feedback with corrected parameter values that remain interpretable as peer neighborhoods change. This lets it contact broader neighborhoods with less communication than the evaluated dense decentralized baselines; as replicas increase, it adds peers while keeping messages sparse, limiting the quality degradation seen with dense gossip. Under bandwidth constraints, the abstract reports higher training throughput than DDP and dense gossip, with throughput sustained during transient link slowdowns.

### Source excerpt

arXiv:2611.23053v1 Announce Type: new Abstract: Pretraining language models across geographically distributed resources requires communication-efficient methods that operate under limited and variable internet bandwidth. Decentralized training replaces global collectives with peer exchanges, but full-model messages make larger peer neighborhoods costly. We introduce SPICE, which combines sparse parameter gossip with error feedback through corrected parameter values that remain directly interpretable as peer neighborhoods change. These sparse exchanges make broader neighborhoods affordable, achieving better training quality with less communication than the evaluated dense decentralized baselines. As the replica count grows, SPICEincreases the number of peers contacted while keeping individual messages sparse, limiting the quality degradation observed with dense gossip. Under bandwidth constraints, SPICE achieves higher training throughput than DDP and dense gossip, while sustaining throughput during transient link slowdowns.

## Gradient Manipulation in Distributed Stochastic Gradient Descent with Strategic Agents: Truthful Incentives with Convergence Guarantees

DevFeed: [Gradient Manipulation in Distributed Stochastic Gradient Descent with Strategic Agents: Truthful Incentives with Convergence Guarantees](<https://devfeed.tech/articles/gradient-manipulation-in-distributed-stochastic-gradient-descent-with-strategic-agents-truthful-incentives-with-convergence-guarantees-141547.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.27962>)

Author: Ziqin Chen, Yongqiang Wang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [behaviors](<https://devfeed.tech/tags/behaviors.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [convex](<https://devfeed.tech/tags/convex.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [incentives](<https://devfeed.tech/tags/incentives.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [manipulate](<https://devfeed.tech/tags/manipulate.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>)

### AI overview

The paper proposes a fully distributed payment mechanism for distributed stochastic gradient descent that aims to incentivize truthful gradient updates while preserving accurate convergence. Unlike prior truthfulness mechanisms described in the abstract, it does not require a centralized server to collect payments or sacrifice convergence accuracy to ensure truthfulness. The authors characterize convergence rates for general convex and strongly convex settings and prove that an agent's cumulative gain from strategic behavior remains finite even over infinitely many iterations. They report experiments on benchmark machine learning tasks; the supplied excerpt gives no specific results or implementation details.

### Source excerpt

arXiv:2603.27962v2 Announce Type: replace Abstract: Distributed learning has gained significant attention due to its advantages in scalability, privacy, and fault tolerance.In this paradigm, multiple agents collaboratively train a global model by exchanging parameters only with their neighbors. However, a key vulnerability of existing distributed learning approaches is their implicit assumption that all agents behave honestly during gradient updates. In real-world scenarios, this assumption often breaks down, as selfish or strategic agents may be incentivized to manipulate gradients for personal gain, ultimately compromising the final learning outcome. In this work, we propose a fully distributed payment mechanism that, for the first time, guarantees both truthful behaviors and accurate convergence in distributed stochastic gradient descent. This represents a significant advancement, as it overcomes two major limitations of existing truthfulness mechanisms for collaborative learning:(1) reliance on a centralized server for payment collection, and (2) sacrificing convergence accuracy to guarantee truthfulness. In addition to characterizing the convergence rate under general convex and strongly convex conditions, we also prove that our approach guarantees the cumulative gain that an agent can obtain through strategic behavior remains finite, even as the number of iterations approaches infinity--a property unattainable by most existing truthfulness mechanisms. Our experimental results on standard machine learning tasks, evaluated on benchmark datasets, confirm the effectiveness of the proposed approach.

## CoreWeave connects custom training loops to model serving

DevFeed: [CoreWeave connects custom training loops to model serving](<https://devfeed.tech/articles/coreweave-connects-custom-training-loops-to-model-serving-138246.md>)

Original publisher: [Read original article](<https://thenewstack.io/coreweave-forge-training-loops/>)

Author: Urvashi Chowdhary

Published: 2026-11-17T16:30:00Z

Content type: article

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-operations](<https://devfeed.tech/tags/ai-operations.md>), [api](<https://devfeed.tech/tags/api.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [coreweave-forge](<https://devfeed.tech/tags/coreweave-forge.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [post-contributed](<https://devfeed.tech/tags/post-contributed.md>), [sponsor-coreweave](<https://devfeed.tech/tags/sponsor-coreweave.md>), [sponsored-post-contributed](<https://devfeed.tech/tags/sponsored-post-contributed.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CoreWeave Forge connects production inference, model training, and serving so teams can turn production failures into evaluated model updates. The excerpt distinguishes distillation for reproducing a stronger model's behavior in a smaller one, supervised fine-tuning when good examples are available, and reinforcement learning when outcomes can be scored but ideal responses are hard to specify; it cautions that weight changes will not fix retrieval, prompts, tools, or serving problems. Model Distillation offers a managed workflow from captured traffic through relabeling and candidate comparison. A programmable Python API lets teams control training logic while CoreWeave manages infrastructure, and RL Rollouts hot-loads checkpoints into live inference while preserving requests and returning training metadata. In a reported Nemotron web-search run with you.com, BrowseComp accuracy rose from 36.97% to 45.45%, average tool calls fell 30.24%, and hot-loading was about 15 times faster than traditional redeployment. The programmable API was in limited preview, with launch planned for later in 2026.

### Source excerpt

Production traffic shows failures that benchmarks or pre-deployment evaluations can't fully reveal. It tells you which edge cases matter and The post CoreWeave connects custom training loops to model serving appeared first on The New Stack.

## TPU Model Design Considerations and Optimization Guide

DevFeed: [TPU Model Design Considerations and Optimization Guide](<https://devfeed.tech/articles/tpu-model-design-considerations-and-optimization-guide-136558.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/tpu-model-design-considerations-and-optimization-guide/>)

Author: Gagik Amirkhanyan; Vaibhav Singh

Published: 2026-11-17T05:43:32.994674Z

Content type: article

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Programming Languages / Compilers](<https://devfeed.tech/topics/programming-languages-compilers.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [arrays](<https://devfeed.tech/tags/arrays.md>), [design](<https://devfeed.tech/tags/design.md>), [developers](<https://devfeed.tech/tags/developers.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [experts](<https://devfeed.tech/tags/experts.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [principles](<https://devfeed.tech/tags/principles.md>), [rules](<https://devfeed.tech/tags/rules.md>), [systolic-arrays](<https://devfeed.tech/tags/systolic-arrays.md>), [tpu](<https://devfeed.tech/tags/tpu.md>)

### AI overview

The guide's core recommendation is to design model shapes, routing, and parallelism around TPU hardware, then validate changes by wall-clock measurements on a tuned stack. Align contracting and output dimensions to multiples of 256; reshape attention to head_dim 256 where quality permits, since smaller dimensions can leave the MXU underused. Keep dynamic workloads in fixed-shape buffers and represent variable extents with metadata to avoid recompilation. For MoEs, LatentMoE compresses routed tokens before dispatch, reducing expert weight reads and communication; a measured 30B model gained 14% end-to-end training throughput with latent width 768. Larger shared experts and fewer, wider routed experts can improve compute intensity, though routing diversity may suffer and the best granularity should be ablated at target scale. Offload sorting and gather/scatter to SparseCore, fuse vector operations where possible, and map communication-heavy parallel axes to fast physical links. The text stresses that measured rankings and gains depend on configuration; thresholds can change across TPU generations.

### Source excerpt

The newly launched AI-Hypercomputer/tpu-codesign guide provides hardware-grounded design rules to help developers maximize performance on TPUs for LLMs and Mixture-of-Experts (MoE) workloads. Key principles include aligning reduction and output dimensions to 256 for systolic arrays, adopting LatentMoE to compress routed token traffic, and mapping parallel axes directly onto physical links.

## Federated versus Centralized PV Capacity-Factor Forecasting: A Non-Inferiority Study Across Seven Stations in North Macedonia

DevFeed: [Federated versus Centralized PV Capacity-Factor Forecasting: A Non-Inferiority Study Across Seven Stations in North Macedonia](<https://devfeed.tech/articles/federated-versus-centralized-pv-capacity-factor-forecasting-a-non-inferiority-study-across-seven-stations-in-north-macedonia-138871.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19421>)

Author: Amar Kumar Mandal, Yuvraj Chaudhary, Dimitar Kitanovski, Eftim Zdravevski

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [factor](<https://devfeed.tech/tags/factor.md>), [federated](<https://devfeed.tech/tags/federated.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [pv](<https://devfeed.tech/tags/pv.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

Across seven North Macedonian stations, the evaluated federated method, FedProx, met the prespecified non-inferiority criterion against centralized training for offline PV capacity-factor forecasting. With a common multilayer perceptron and 24-hour timestamp lag, FedProx had macro-MAE 0.128397 versus 0.133405; the paired difference was -0.005008 (95% CI -0.009926 to -0.000537), below the 0.005 margin. All five seed comparisons favored FedProx, and its mean MAE was lower at six stations. The comparison used past-only inputs and 49 test dates; calendar sensitivity analyses did not change the conclusion. This supports further evaluation of federated forecasting where raw data cannot be pooled. Benefits for energy commitments or battery scheduling remain potential applications requiring operational evaluation.

### Source excerpt

arXiv:2611.19421v1 Announce Type: new Abstract: Background: Photovoltaic (PV) forecasts inform energy commitments by PV operators and aggregators and guide charge-discharge scheduling in solar-plus-storage systems. Federated learning enables collaborative model training through model updates when local operators cannot pool raw operational data. This study evaluated whether federated training was non-inferior to centralized training for PV capacity-factor prediction across seven stations in North Macedonia. A common multilayer perceptron was used in an offline forecasting comparison with a 24-hour timestamp lag. Exploratory validation compared local, centralized, Federated Averaging, and Federated Proximal (FedProx) training. FedProx (E = 1, mu = 1) was selected for a confirmatory comparison under a protocol fixed before test evaluation. The comparison used past-only offline inputs, five paired training seeds, macro-averaged mean absolute error (MAE), a prespecified non-inferiority margin of 0.005, and a paired day bootstrap with 10,000 repetitions over 49 test dates. Results: FedProx achieved a macro-MAE of 0.128397, compared with 0.133405 for centralized training. The paired difference (FedProx minus centralized) was -0.005008 (95% confidence interval: [-0.009926, -0.000537]). The upper confidence bound was below the prespecified margin, satisfying the non-inferiority criterion. All five seed-specific differences favored FedProx, and its mean MAE was lower at six of the seven stations. Post hoc calendar sensitivity analyses did not change the conclusion. Conclusions: These findings support further evaluation of federated forecasting for collaboration under data-pooling constraints. The study demonstrates predictive non-inferiority within the evaluated setting. Benefits for energy commitments and battery scheduling remain potential applications requiring operational evaluation.

## Supercharging Olmo-core for Efficient and Scalable MoE Training

DevFeed: [Supercharging Olmo-core for Efficient and Scalable MoE Training](<https://devfeed.tech/articles/supercharging-olmo-core-for-efficient-and-scalable-moe-training-136978.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18956>)

Author: Tianhua Tao, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas Samaga, Jacob Morrison, Iz Beltagy, Taira Anderson, Noah A. Smith, Hannaneh Hajishirzi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [b300](<https://devfeed.tech/tags/b300.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [ddp](<https://devfeed.tech/tags/ddp.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [fsdp](<https://devfeed.tech/tags/fsdp.md>), [gerrymandering](<https://devfeed.tech/tags/gerrymandering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvl8](<https://devfeed.tech/tags/nvl8.md>), [nvshmem](<https://devfeed.tech/tags/nvshmem.md>), [pipeline-parallelism](<https://devfeed.tech/tags/pipeline-parallelism.md>), [router](<https://devfeed.tech/tags/router.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Olmo-core's redesigned MoE training stack targets costs that grow with stored expert parameters even when per-token compute stays nearly constant. It replaces earlier FSDP with DDP plus expert and pipeline parallelism, a distributed optimizer, and activation recompute. NVSHMEM-based rowwise expert parallelism writes tokens directly to expert buffers; GPU-resident routing metadata and device-scheduled grouped GEMMs remove synchronization. Integrated MXFP8 reduces compute, activation-memory, and transport costs, while topology-agnostic checkpoints allow parallel layouts to change between runs. On NVL8 B300 nodes, configurations span 12.9 billion to 1.2 trillion parameters across up to 512 GPUs; the 1.2-trillion configuration reaches 858 useful-model TFLOP/s/GPU with MXFP8 and per-layer recompute. An experimental test with optional DeepEP v2 reaches 2.38 trillion parameters. The authors also report router exploitation of the load-balancing loss ("Token Gerrymandering") and warn that communication overlap can slow kernels more than it hides.

### Source excerpt

arXiv:2611.18956v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismatch is costly: in early baselines, throughput fell as stored expert parameter size grew, even though the useful work per token stayed nearly fixed. To close this gap, we present a redesigned training stack in Olmo-core, built for efficient, scalable MoE training. We replace the Fully Sharded Data Parallel (FSDP) of earlier Olmo-core versions with an architecture built on Distributed Data Parallel (DDP), and add Expert Parallelism (EP), Pipeline Parallelism (PP), a distributed optimizer, and activation recompute. We propose NVSHMEM-based rowwise EP, which writes each token directly into its expert's buffer, avoiding dispatch and combine permutation overhead. We make EP fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs. Closely integrated MXFP8 cuts compute, activation-memory, and transport costs, and topology-agnostic checkpoints let the parallel layout change between runs. On NVL8 B300 nodes, the stack supports training configurations from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, reaching 858 useful-model TFLOP/s/GPU at 1.2 trillion with MXFP8 and per-layer recompute; an experimental capacity test with the optional DeepEP v2 backend reaches 2.38 trillion. We also explain each design choice and share findings for other training systems; for example, we identify a failure pattern we call Token Gerrymandering, in which the MoE router can learn to hack the load-balancing loss, and find that overlapping communication with computation can slow the overlapped kernels by more than it hides.

## ReCALM: A Residual-enhanced Crowdsourced Analytic Learning Mechanism for Distributed Continual Service Construction

DevFeed: [ReCALM: A Residual-enhanced Crowdsourced Analytic Learning Mechanism for Distributed Continual Service Construction](<https://devfeed.tech/articles/recalm-a-residual-enhanced-crowdsourced-analytic-learning-mechanism-for-distributed-continual-service-construction-138849.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19276>)

Author: Kejia Fan, Yajiang Huang, Jianheng Tang, Jinfeng Xu, Huiping Zhuang, Anfeng Liu, Houbing Herbert Song, Yunhuai Liu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [construction](<https://devfeed.tech/tags/construction.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cross](<https://devfeed.tech/tags/cross.md>), [crowdsourcing](<https://devfeed.tech/tags/crowdsourcing.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [deep](<https://devfeed.tech/tags/deep.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [model](<https://devfeed.tech/tags/model.md>), [service](<https://devfeed.tech/tags/service.md>)

### AI overview

ReCALM targets catastrophic forgetting in continual service construction across heterogeneous worker data. It extends gradient-free analytic learning beyond single-layer linear models with residual blocks that have closed-form solutions, and uses a cross-recursive least-squares strategy for distributed continual training. The abstract reports that a 20-layer version improves accuracy by 8.85%-9.83% over a single-layer analytic baseline and reduces computational costs by over 94% compared with gradient-based baselines. These are reported results; the excerpt provides no further experimental details.

### Source excerpt

arXiv:2611.19276v1 Announce Type: new Abstract: Crowdsourcing has emerged as a promising paradigm for constructing AI services by harnessing the collective capabilities of distributed workers. As consumers' demands continually evolve, service models must acquire new knowledge while retaining previously learned information. However, due to the inherited reliance on gradients for backpropagation, existing approaches typically suffer from catastrophic forgetting during continual model updates on heterogeneous data. Recent studies have introduced analytic learning, a gradient-free paradigm that achieves invariance through recursive closed-form solutions. Nevertheless, existing analytic methods are typically constrained by single-layer linear networks, which tend to suffer from underfitting due to their inability to perform deep representation learning. To address this issue, we propose ReCALM, a Residual-enhanced Crowdsourced Analytic Learning Mechanism that achieves deep nonlinear invariance for distributed continual service construction. Inspired by the success of ResNet, we develop gradient-free residual blocks for ReCALM with closed-form solutions. To implement distributed continual training for ReCALM, we further devise a cross-recursive strategy via least squares. Both theoretical and experimental analyses verify ReCALM's dual superiority in heterogeneity invariance and deep representation. With 20 residual layers, ReCALM improves accuracy by 8.85%-9.83% over the single-layer analytic baseline, while reducing computational costs by over 94% compared with gradient-based baselines.

## CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1

DevFeed: [CoreWeave Becomes the First Cloud Provider to Deliver Multi-Rack NVIDIA Vera Rubin Results in MLPerf® Training v6.1](<https://devfeed.tech/articles/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-training-v6-1-136727.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/coreweave-becomes-the-first-cloud-provider-to-deliver-multi-rack-nvidia-vera-rubin-results-in-mlperf-r-training-v6-1>)

Author: CoreWeave

Published: 2026-11-16T19:52:27Z

Content type: news

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [NVIDIA Vera Rubin](<https://devfeed.tech/topics/nvidia-vera-rubin.md>), [Vera Rubin NVL72](<https://devfeed.tech/topics/vera-rubin-nvl72.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [NVIDIA Spectrum-X Ethernet](<https://devfeed.tech/topics/nvidia-spectrum-x-ethernet.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-provider](<https://devfeed.tech/tags/cloud-provider.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [delivered](<https://devfeed.tech/tags/delivered.md>), [first](<https://devfeed.tech/tags/first.md>), [full](<https://devfeed.tech/tags/full.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [leading](<https://devfeed.tech/tags/leading.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-vera](<https://devfeed.tech/tags/nvidia-vera.md>), [nvidia-vera-rubin](<https://devfeed.tech/tags/nvidia-vera-rubin.md>), [rack](<https://devfeed.tech/tags/rack.md>), [results](<https://devfeed.tech/tags/results.md>), [rubin](<https://devfeed.tech/tags/rubin.md>), [training](<https://devfeed.tech/tags/training.md>), [vera](<https://devfeed.tech/tags/vera.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>)

### AI overview

In MLPerf Training v6.1, CoreWeave reports training DeepSeek-V3 671B to target quality in 17.65 minutes on 256 Vera Rubin GPUs across four racks, versus 48.40 minutes on the same number of GB200 GPUs (2.7x faster, per CoreWeave's calculation). It also reports the round's fastest 72-GPU Llama 3.1 8B submission: 7.16 minutes on one Vera Rubin rack, compared with 13.70 minutes on GB200. The article attributes multi-rack performance to an integrated stack: Spectrum-X RoCE networking and ConnectX-9 SuperNICs, topology-aware placement and orchestration, optimized software, and fleet monitoring and recovery. On GB200, it reports Qwen3.5 397B reinforcement learning in 22.78 minutes across four racks, split evenly between training and generation using NeMo and TensorRT-LLM. These are CoreWeave's descriptions of published results; its speedup and fastest-submission claims are its own analyses and were not verified by MLCommons.

### Source excerpt

See how CoreWeave's full-stack optimizations delivered leading NVIDIA Vera Rubin performance in MLPerf® Training v6.1, from a single 72-GPU rack to multi-rack scale at 256 GPUs

## New AI training performance benchmarks for Google TPUs

DevFeed: [New AI training performance benchmarks for Google TPUs](<https://devfeed.tech/articles/new-ai-training-performance-benchmarks-for-google-tpus-137754.md>)

Original publisher: [Read original article](<https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/training-performance-benchmarks-tpus/>)

Author: Google Threat Intelligence Group

Published: 2026-11-16T16:00:00Z

Content type: news

Language: en

Sources: [Google Cloud](<https://devfeed.tech/sources/google-cloud.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [chips](<https://devfeed.tech/tags/chips.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [grace-blackwell](<https://devfeed.tech/tags/grace-blackwell.md>), [latest](<https://devfeed.tech/tags/latest.md>), [mlperf](<https://devfeed.tech/tags/mlperf.md>), [none](<https://devfeed.tech/tags/none.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [performance](<https://devfeed.tech/tags/performance.md>), [results](<https://devfeed.tech/tags/results.md>), [tpus](<https://devfeed.tech/tags/tpus.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Google Cloud reports 10 MLPerf Training v6.1 results spanning Ironwood TPUs and NVIDIA Grace Blackwell GPUs. On Ironwood, 8,192 TPUs reached the pre-training target quality in 1.70 minutes, with scaling to 4,096 chips reported at 96% efficiency. For the first MLPerf agentic reinforcement learning benchmark, 640 Ironwood TPUs reached 0.765 accuracy against a 0.69 goal in 18.02 minutes; code generation and learning ran simultaneously, and training chips waited less than 3% of the time. On Google Cloud A4X Max Instances, 256 NVIDIA GB300 GPUs reached pre-training target quality in 31.96 minutes. Google also submitted five A4X and A4X Max results for agentic RL. These are vendor-reported benchmark figures from the supplied excerpt.

### Source excerpt

In the latest MLPerf Training v6.1 benchmark round, Google Cloud published 10 results across seventh-generation Ironwood TPUs and NVIDIA Grace Blackwell GPUs, spanning f...

## D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models

DevFeed: [D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models](<https://devfeed.tech/articles/d-vla-a-high-concurrency-distributed-asynchronous-reinforcement-learning-framework-for-vision-language-action-models-134982.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.13276>)

Author: Yucheng Guo, Yongjian Guo, Zhong Guan, Wen Huang, Haodong Yue, Shuai Di, Junwu Xiong, Yicheng Gong, Xi Xiao

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [communication](<https://devfeed.tech/tags/communication.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [data](<https://devfeed.tech/tags/data.md>), [decoupling](<https://devfeed.tech/tags/decoupling.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embodied-ai](<https://devfeed.tech/tags/embodied-ai.md>), [environments](<https://devfeed.tech/tags/environments.md>), [execution](<https://devfeed.tech/tags/execution.md>), [framework](<https://devfeed.tech/tags/framework.md>), [learning](<https://devfeed.tech/tags/learning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [vision-language-action](<https://devfeed.tech/tags/vision-language-action.md>), [vision-language-action-models](<https://devfeed.tech/tags/vision-language-action-models.md>), [vla](<https://devfeed.tech/tags/vla.md>)

### AI overview

D-VLA addresses throughput bottlenecks in distributed reinforcement learning for large Vision-Language-Action models by separating high-frequency training data from low-frequency weight control and overlapping sampling, inference, gradient computation, and parameter distribution in a four-thread asynchronous pipeline. It also uses dual-pool VRAM management and topology-aware replication to address memory fragmentation and communication efficiency. The abstract reports higher throughput and sampling efficiency than mainstream RL frameworks on LIBERO for billion-parameter models, and stable linear speedup in trillion-parameter scalability tests.

### Source excerpt

arXiv:2605.13276v3 Announce Type: replace Abstract: The rapid evolution of Embodied AI has enabled Vision-Language-Action (VLA) models to excel in multimodal perception and task execution. However, applying Reinforcement Learning (RL) to these massive models in large-scale distributed environments faces severe systemic bottlenecks, primarily due to the resource conflict between high-fidelity physical simulation and the intensive VRAM/bandwidth demands of deep learning. This conflict often leaves overall throughput constrained by execution-phase inefficiencies. To address these challenges, we propose D-VLA, a high-concurrency, low-latency distributed RL framework for large-scale embodied foundation models. D-VLA introduces "Plane Decoupling," physically isolating high-frequency training data from low-frequency weight control to eliminate interference between simulation and optimization. We further design a four-thread asynchronous "Swimlane" pipeline, enabling full parallel overlap of sampling, inference, gradient computation, and parameter distribution. Additionally, a dual-pool VRAM management model and topology-aware replication resolve memory fragmentation and optimize communication efficiency. Experiments on benchmarks like LIBERO show that D-VLA significantly outperforms mainstream RL frameworks in throughput and sampling efficiency for billion-parameter VLA models. In trillion-parameter scalability tests, our framework maintains exceptional stability and linear speedup, providing a robust system for high-performance general-purpose embodied agents.

## FedSharp: Byzantine Robust Decentralized Federated Learning Under Noisy Channel

DevFeed: [FedSharp: Byzantine Robust Decentralized Federated Learning Under Noisy Channel](<https://devfeed.tech/articles/fedsharp-byzantine-robust-decentralized-federated-learning-under-noisy-channel-135738.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05546>)

Author: Ojaswi Sinha, Amin Farajzadeh, Animesh Yadav, Melike Erol-Kantarci

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cifar-10](<https://devfeed.tech/tags/cifar-10.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [learning](<https://devfeed.tech/tags/learning.md>), [median](<https://devfeed.tech/tags/median.md>), [mnist](<https://devfeed.tech/tags/mnist.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [node](<https://devfeed.tech/tags/node.md>), [noise](<https://devfeed.tech/tags/noise.md>), [ring](<https://devfeed.tech/tags/ring.md>), [robust](<https://devfeed.tech/tags/robust.md>), [screens](<https://devfeed.tech/tags/screens.md>)

### AI overview

FedSharp targets decentralized federated learning's vulnerability to Byzantine nodes, non-IID data, and channel noise that accumulates as models pass between nodes. Each node retains models from its 2b predecessors, providing at least b honest candidates when at most b nodes are Byzantine; it screens by deviation from the coordinate-wise median, averages the b lowest-loss candidates to reduce fresh noise variance, and uses hop-aware gradient-norm regularization and proximal training to address accumulated noise and data drift. In tests on a 100-node ring with 33 Byzantine nodes, it reached 93.5-94.9% accuracy on MNIST and 32.8-48.8% on CIFAR-10 across Gaussian and hidden attacks under IID and non-IID data, outperforming the reported single-model-selection benchmark. On MNIST, it came within 1.3 percentage points of the no-channel-noise benchmark and exceeded it under the hidden attack.

### Source excerpt

arXiv:2611.05546v2 Announce Type: replace Abstract: Decentralized federated learning (DFL) eliminates the central aggregation server but remains vulnerable to non-independent and identically distributed (non-IID) data, Byzantine attacks, and channel noise. We show that channel noise accumulates as models propagate across nodes and becomes increasingly significant as the learning rate decays. Moreover, accumulated noise enables Byzantine attackers to exploit loss-based model selection by mimicking the average of the honest models. To address these challenges, we propose FedSharp (screened, hop-aware, averaged, ranked, proximal), a robust DFL framework. In FedSharp, each node keeps the latest models received from its 2b predecessors, which guarantees at least b honest candidates when at most b nodes are Byzantine. FedSharp screens suspicious models using their deviations from the coordinate-wise median, averages the b lowest-loss candidates to reduce the fresh channel-noise variance by a factor of b, and applies hop-aware gradient-norm regularization and proximal training to mitigate accumulated noise and non-IID drift. Experiments on a 100-node ring with 33 Byzantine nodes show that FedSharp achieves 93.5 to 94.9 percent accuracy on MNIST, compared with 29.4 to 38.1 percent for a state-of-the-art single-model-selection benchmark, and 32.8 to 48.8 percent on CIFAR-10, compared with 10.0 to 30.5 percent, across Gaussian and hidden attacks under IID and non-IID data. On MNIST, FedSharp comes within 1.3 percentage points of the benchmark operating without channel noise, and exceeds it under the hidden attack.

## Federated Physics-Informed Thermal Field Prediction Across Unseen Geometries in Distributed Additive Manufacturing

DevFeed: [Federated Physics-Informed Thermal Field Prediction Across Unseen Geometries in Distributed Additive Manufacturing](<https://devfeed.tech/articles/federated-physics-informed-thermal-field-prediction-across-unseen-geometries-in-distributed-additive-manufacturing-135832.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17794>)

Author: Amirul Islam Saimon (James), Zhenyu (James), Kong

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>)

Tags: [2d](<https://devfeed.tech/tags/2d.md>), [additive-manufacturing](<https://devfeed.tech/tags/additive-manufacturing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [field](<https://devfeed.tech/tags/field.md>), [inverse-design](<https://devfeed.tech/tags/inverse-design.md>), [manufacturing](<https://devfeed.tech/tags/manufacturing.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [physics-informed](<https://devfeed.tech/tags/physics-informed.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [thermal](<https://devfeed.tech/tags/thermal.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

### AI overview

Fed-PIGANO trains physics-informed geometry-aware neural operator models at separate additive manufacturing sites and aggregates their encoders and prediction networks without sharing proprietary part geometries. On a 2D steady-state metal AM benchmark, geometries from five polygonal families were split across three sites in four scenarios, from matching to strongly different family mixes. Across seven replications, mean relative L2 error was 1.83%-2.04% in all 12 scenario-site cases, versus as high as 20.46% for local-only training; PDE residuals were also consistently lower. Accuracy was comparable to a centralized model trained on pooled geometries. The evidence is limited to this benchmark and supports potential use for predicting thermal fields on new designs, including shape optimization, inverse design, and uncertainty quantification.

### Source excerpt

arXiv:2611.17794v1 Announce Type: new Abstract: In additive manufacturing (AM), each change in part geometry requires a new thermal simulation, which makes design evaluation over many candidates costly. Geometry-aware models reduce this cost by predicting thermal fields on unseen parts with a single learned model, but they assume that diverse geometries are available centrally. This assumption is restrictive when each manufacturing site holds only a limited set of proprietary part designs that it cannot share. We propose Fed-PIGANO, a federated framework that trains physics-informed geometry-aware neural operator (PI-GANO) models at individual sites and aggregates their geometry encoders and prediction networks without sharing part geometries. We evaluate the framework on a 2D steady-state thermal benchmark for metal AM, with geometries from five polygonal families distributed across three sites under four scenarios that range from identical to strongly different family compositions. Over seven replications, Fed-PIGANO achieves mean relative L2 errors of 1.83% to 2.04% across all 12 scenario-site cases, compared with up to 20.46% for local-only training, and consistently yields lower PDE residuals. Its prediction accuracy is comparable to that of a centralized model trained on the pooled geometries of all sites. These results demonstrate that federated learning lets manufacturing sites benefit from each other's geometries without sharing them, enabling rapid thermal prediction on new part designs for tasks such as shape optimization, inverse design, and uncertainty quantification.

## HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs

DevFeed: [HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs](<https://devfeed.tech/articles/hetccl-efficient-llm-training-on-heterogeneous-vendor-gpus-134880.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2601.22585>)

Author: Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [llm-training](<https://devfeed.tech/tags/llm-training.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [rdma](<https://devfeed.tech/tags/rdma.md>)

### AI overview

HetCCL enables collective communication across NVIDIA and AMD GPUs using RDMA, without driver modifications or changes to existing deep learning applications. It unifies vendor-specific backends and introduces two mechanisms that leverage NVIDIA NCCL and AMD RCCL. In the reported multi-vendor cluster evaluation, HetCCL matched NCCL and RCCL performance in homogeneous setups and scaled in heterogeneous environments. The supplied text does not describe the mechanisms or provide detailed evaluation results.

### Source excerpt

arXiv:2601.22585v2 Announce Type: replace Abstract: The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.

## BOA Constrictor: Squeezing Performance out of GPUs in the Cloud via Budget-Optimal Allocation

DevFeed: [BOA Constrictor: Squeezing Performance out of GPUs in the Cloud via Budget-Optimal Allocation](<https://devfeed.tech/articles/boa-constrictor-squeezing-performance-out-of-gpus-in-the-cloud-via-budget-optimal-allocation-131508.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.01404>)

Author: Zhouzi Li, Cindy Zhu, Arpan Mukhopadhyay, Mor Harchol-Balter, Benjamin Berg

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>)

Tags: [1-6](<https://devfeed.tech/tags/1-6.md>), [2](<https://devfeed.tech/tags/2.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [simulations](<https://devfeed.tech/tags/simulations.md>)

### AI overview

BOA Constrictor schedules cloud GPUs under a fixed budget to minimize average job completion time. Its Budget-Optimal Allocation policy jointly decides how many GPUs to rent over time and how to distribute them among active training jobs, balancing faster completion against GPU-hour costs and diminishing gains from adding GPUs. The abstract reports average JCT reductions of 1.6x in small-scale implementation experiments and 2x in large-scale simulations, compared with state-of-the-art heuristic schedulers, at a given budget.

### Source excerpt

arXiv:2602.01404v3 Announce Type: replace Abstract: The past decade has seen a dramatic increase in demand for GPUs to train Machine Learning (ML) models. Because it is prohibitively expensive for most organizations to build and maintain a large GPU cluster, organizations instead choose to rent GPUs from cloud providers. The customer is responsible for devising a policy for (i) deciding how many GPUs to rent at every moment in time to process a stream of ML training jobs and (ii) allocating the rented GPUs among the currently active jobs in the system. Because ML training jobs can be parallelized across different numbers of GPUs, the customer generally has many options for how many GPUs to use for each job. Allocating more GPUs to a single training job will cause the job to complete more quickly. However, the customer pays for each GPU-hour they use, and a training job receives a diminishing marginal benefit from running on additional GPUs. Hence, allocating too many GPUs to a single training job can dramatically increase the overall cost that the customer pays to the cloud provider. This gives rise to a cost-performance tradeoff that customers must balance when running training jobs in the cloud. To balance the cost-performance tradeoff, we develop BOA Constrictor, a new scheduler for ML training jobs which uses a Budget-Optimal Allocation (BOA) policy to squeeze the highest level of performance out of a cloud-deployed GPU cluster given a fixed budget constraint. We explicitly formulate the problem as a budget-constrained scheduling problem and derive the BOA policy which minimizes the average job completion time (JCT) of a stream of arriving jobs subject to the user's budget. For a given budget level, we demonstrate that BOA Constrictor can reduce average JCT by 1.6 times in small-scale implementation experiments and by 2 times in detailed, large-scale simulations compared to state-of-the-art heuristic based schedulers.

## Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

DevFeed: [Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training](<https://devfeed.tech/articles/beyond-the-trial-and-error-loop-hybrid-projection-and-automated-tuning-for-distributed-training-131516.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.19423>)

Author: Anshu Raina, Peyman Razaghi, Yuankai Chen, Cheng Yao, Yao Fu, Vidushi Goyal, Devang Patel, Wen Xie, Zhenyu Gu, Emad Barsoum

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [automated](<https://devfeed.tech/tags/automated.md>), [axis](<https://devfeed.tech/tags/axis.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [crashes](<https://devfeed.tech/tags/crashes.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [deepseek-v3](<https://devfeed.tech/tags/deepseek-v3.md>), [device](<https://devfeed.tech/tags/device.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [engine](<https://devfeed.tech/tags/engine.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [fast](<https://devfeed.tech/tags/fast.md>), [filter](<https://devfeed.tech/tags/filter.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [projection](<https://devfeed.tech/tags/projection.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Primus Projection estimates distributed-training throughput by combining GPU microbenchmarks with analytical scaling of collectives, FSDP overlap, and pipeline schedules; it also offers CPU-only simulation for planning without GPUs. The Primus Tuning Agent uses these projections to score candidates in seconds, combining single-axis sweeps with LLM-proposed cross-axis configurations and a memory-feasibility filter that rejects invalid candidates before GPU runs. Across the reported models and two GPU generations, projections stayed within about 10% of measured throughput, including 4.7% error at 1,024 GPUs and 1.9% at 256 GPUs. In a Mixtral 8x22B case study, tuning found a configuration with 27% higher throughput than the cited four-node BF16 reference in under 30 minutes of single-node exploration.

### Source excerpt

arXiv:2610.19423v2 Announce Type: replace-cross Abstract: Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.

## LoCoJAX: Host-Local Coordination for Small-Model RL on TPUs

DevFeed: [LoCoJAX: Host-Local Coordination for Small-Model RL on TPUs](<https://devfeed.tech/articles/locojax-host-local-coordination-for-small-model-rl-on-tpus-131546.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16301>)

Author: Loki Chen, Gregory B. Rehm, Yang Song, Hongye Xie, Ming Lei, Chongyang Bai, Bob Kamma, Mingfu Liang, Kavosh Asadi, Yufei Li, Xi Liu, Jay Xu, Rui Liu, Maziar Sanjabi, Luke Simon, Hamed Firooz

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [faster](<https://devfeed.tech/tags/faster.md>), [jax](<https://devfeed.tech/tags/jax.md>), [local](<https://devfeed.tech/tags/local.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [ray](<https://devfeed.tech/tags/ray.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rl](<https://devfeed.tech/tags/rl.md>), [separate](<https://devfeed.tech/tags/separate.md>), [small](<https://devfeed.tech/tags/small.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [small-models](<https://devfeed.tech/tags/small-models.md>), [tpus](<https://devfeed.tech/tags/tpus.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

On TPU v7x, host orchestration can dominate coordination overhead for small-model RL: in a decode-heavy sweep, it takes 36% of step time for a 0.6B model, while network communication takes 2%; for large models, estimated network costs dominate non-compute time. LoCoJAX keeps rollout and reward processing local to each host while training synchronously across hosts. In comparisons with a tuned Pathways baseline on Qwen3-0.6B, its colocated mode runs steps 1.3-3.6x faster across three workloads and 8-64 chips; on GSM8K, 8 chips beat Pathways on 64. Its Ray-managed disaggregated mode overlaps rollout and training, but helps only when rollout dominates and receives most hosts. On NuminaMath-CoT at 256 chips, assigning rollout 75% of hosts reduced step time about 15% versus colocated mode.

### Source excerpt

arXiv:2611.16301v1 Announce Type: cross Abstract: Reinforcement learning (RL) increasingly targets small language models, and executing these workloads efficiently across multiple accelerators requires effective coordination. On TPUs, Pathways-based RL uses a single controller to coordinate computation across hosts. This execution model is effective for large models, but its coordination computational costs are harder to amortize when small models repeatedly invoke short computations. To characterize these computational costs, we sweep dense models from 0.6B to 405B parameters on TPU v7x under a decode-heavy workload and measure how host and network computational overhead vary with model size. At 0.6B, host orchestration occupies 36% of step time and network communication 2%, whereas for large models, estimated network computational costs dominate the non-compute time. Motivated by these findings, we present Local Coordination JAX (LoCoJAX), which implements local rollout and reward processing on each TPU host with synchronous global training. In controlled comparisons against a tuned and optimized Pathways baseline on Qwen3-0.6B across three workloads and 8-64 TPU v7x chips, its colocated mode (LoCoJAX-CO) is 1.3-3.6x faster per step; on GSM8K, it completes a step faster on 8 chips than Pathways does on 64. Finally, a Ray-managed disaggregated mode (LoCoJAX-DA) overlaps rollout and training on separate hosts, and placement experiments establish when this overlap compensates for dividing devices and transferring weights: only when rollout dominates the step and receives most of the hosts. On NuminaMath-CoT at 256 chips, giving rollout 75% of hosts cuts step time by about 15% compared with LoCoJAX-CO.

## Cortex-FL: Agent-Mediated Collaboration for Federated Learning

DevFeed: [Cortex-FL: Agent-Mediated Collaboration for Federated Learning](<https://devfeed.tech/articles/cortex-fl-agent-mediated-collaboration-for-federated-learning-132319.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17186>)

Author: Rui Sun, Jinfeng Peng, Zehua Cheng, Jiahao Sun, Zhipeng Wang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [blockchain](<https://devfeed.tech/tags/blockchain.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [control](<https://devfeed.tech/tags/control.md>), [cortex](<https://devfeed.tech/tags/cortex.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [data](<https://devfeed.tech/tags/data.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [execution](<https://devfeed.tech/tags/execution.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [financial](<https://devfeed.tech/tags/financial.md>), [incentives](<https://devfeed.tech/tags/incentives.md>), [independent](<https://devfeed.tech/tags/independent.md>), [learning](<https://devfeed.tech/tags/learning.md>)

### AI overview

Cortex-FL uses agents to handle the coordination around federated learning: they turn natural-language objectives into shared tasks, find collaborators, and negotiate participation agreements, then connect those agreements to distributed training and incentive management. Owners delegate coordination while retaining authority over local data, computation, and financial commitments. The abstract reports a complete four-node demonstration, from a natural-language model request through contract-governed training and verified delivery to x402 settlement on a blockchain testnet; it does not establish performance beyond that demonstration.

### Source excerpt

arXiv:2611.17186v1 Announce Type: new Abstract: Federated learning (FL) enables independent data owners to train models collaboratively without centralizing raw data. However, identifying collaborators, preparing training tasks, and coordinating execution and incentives hinder its practical adoption by organizations without specialized expertise. We present Cortex-FL, an agent-native FL system that enables users to express learning objectives in natural language and delegate the collaboration required to train a model. Agents translate these objectives into shared tasks, discover collaborators, and negotiate participation agreements, connecting collaboration formation with distributed training and incentive management. The architecture separates agent coordination from local execution authority, allowing owners to delegate coordination while retaining control over their data, computation, and financial commitments. We demonstrate the complete workflow on a four-node deployment, in which a model requested in natural language proceeds through collaboration formation, contract-governed training, verified model delivery, and x402 settlement on a blockchain testnet.

[Next page](<https://devfeed.tech/topics/distributed-training.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgIjVjMDRjMjk5LWU3YzMtNDQ1NC05YmI0LWRkMzIwMzcyNTQ1YyJd>)