# Mixture of Experts (MoE) A machine learning technique that combines specialized expert models, with a gating network activating experts for an input. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each DevFeed: [Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-15T17:00:11Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Mixture of Experts (MoE)](), [Nemotron](), [Routing (disambiguation)]() Tags: [agentic-ai-generative-ai](), [ai-inference](), [architecture](), [cost](), [data-center-cloud](), [deployment](), [llms](), [memory](), [mixture-of-experts-moe](), [models](), [moe](), [nemo](), [nemotron](), [performance](), [router](), [routing](), [tokens]() ### AI overview This article explains how dense and Mixture-of-Experts models activate parameters, compares their effects on throughput, memory cost, and serving complexity, and discusses when each architecture fits different deployment constraints. It uses Nemotron 3.5 Lightning as an example of an MoE model. ### Source excerpt How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the... ## Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine DevFeed: [Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-14T16:39:15Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Mixture of Experts (MoE)](), [Artificial Intelligence](), [GPU](), [Large Language Model](), [Nvidia](), [Routing (disambiguation)](), [GB200](), [Network](), [Python](), [deepseek](), [mixtral](), [qwen]() Tags: [agentic-ai-generative-ai](), [ai](), [communication](), [deepseek](), [developer-tools-techniques](), [gb200](), [gpu](), [mixtral](), [mixture-of-experts-moe](), [mlops](), [moe](), [networks](), [nvfp4](), [nvidia](), [performance](), [python](), [qwen](), [router](), [routing](), [tensors](), [token](), [tokens](), [training-ai-models]() ### AI overview This article explains how NVIDIA Transformer Engine and JAX kernel optimizations accelerate dropless Mixture of Experts (MoE) training. It describes bottlenecks from token routing, expert dispatch and gathering, all-to-all communication, and ragged expert matrix operations. In DeepSeek-V3 training on NVIDIA GB200, the optimized approach increased performance from 103 to 1,068 TFLOPS per GPU, a 10.4x improvement. ### Source excerpt Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE... ## "Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason DevFeed: ["Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason]() Original publisher: [Read original article]() Author: Adrian Bridgwater Published: 2026-09-13T14:21:46Z Content type: news Language: en Sources: [The New Stack]() Topics: [Mixture of Experts (MoE)](), [cohere](), [Artificial Intelligence](), [Benchmark](), [Inference](), [qwen](), [gemma4](), [Google]() Tags: [ai](), [ai-infrastructure](), [ai-models](), [aya](), [benchmark](), [cohere](), [gemma](), [google](), [inference](), [large-language-models](), [mixture-of-experts-moe](), [model](), [open](), [qwen]() ### AI overview Cohere's North Small Translate is an open-weight mixture-of-experts machine translation model covering 50 languages. The article discusses its non-reasoning design, sovereign AI positioning, deployment options, efficiency claims, and reported WMT26 benchmark comparisons. ### Source excerpt Enterprise AI company Cohere announced North Small Translate last week, a mixture-of-experts (MOE) open-weight machine translation model that works across The post "Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason appeared first on The New Stack. ## Открываем претрейн Alice AI Search: как устроена модель быстрых ответов Алисы на Поиске DevFeed: [Открываем претрейн Alice AI Search: как устроена модель быстрых ответов Алисы на Поиске]() Original publisher: [Read original article]() Author: pet67 (Яндекс) Published: 2026-09-11T06:05:13Z Content type: article Language: ru Sources: [Яндекс - Как мы делаем Яндекс / Статьи]() Topics: [Artificial Intelligence](), [AI search](), [Transformers](), [Mixture of Experts (MoE)](), [Large Language Model](), [hugging face]() Tags: [agentic](), [ai](), [ai-search](), [alice-ai](), [hugging-face](), [llm](), [ml](), [moe](), [rl](), [tag-178bc8f01f24](), [tag-4004cf5948d3](), [tag-61cd5a476b1d](), [tag-d89cae10e887](), [tag-e6d9cc1f0757](), [transformers]() ### AI overview This developer article explains the Alice AI Search pipeline for generating fast answers, including its search and context-processing stages, shorter information contexts, a sparse Mixture-of-Experts architecture combined with an Encoder-Decoder, and online reinforcement learning from user behavior signals. It also announces the open release of the Alice AI-T5-35B-A0.6B Base model, with external inference available through Hugging Face Transformers while optimized production inference remains internal to Yandex. ### Source excerpt Быстрый ответ Алисы AI -- это самый массовый генеративный продукт Яндекса и первое соприкосновение с Алисой для пользователей Поиска. Даже в час пиковой нагрузки пользователь должен получить лаконичный ответ за считаные секунды. Для этого мы, команда Alice AI Search, адаптируем весь пайплайн быстрых ответов -- от собственного претрейна с кастомной архитектурой до онлайн-rl-обучения на поведенческие сигналы пользователей. В статье разберём, как устроен генеративный ответ в Поиске, и расскажем про основные улучшения июньского релиза: как мы ускорили ответы за счёт коротких инфоконтекстов, зачем совместили Encoder-Decoder с разреженной MoE-архитектурой и как обучение на реальных пользовательских сигналах повлияло на качество и использование продукта. Кроме того, мы выложили в открытый доступ обученную с нуля модель Alice AI-T5-35B-A0.6B Base с тем ограничением, что внешним пользователям доступен инференс через Hugging Face Transformers, а оптимизированный production-инференс пока доступен только внутри Яндекса. Читать далее ## Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM DevFeed: [Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM]() Original publisher: [Read original article]() Author: Dmitry Soldatkin Published: 2026-09-09T22:26:29Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [Deployment](), [Inference](), [Machine learning](), [Language models](), [GPU](), [long-context](), [Mixture of Experts (MoE)]() Tags: [advanced-300](), [agentic](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [deployment](), [gpu](), [how-to](), [inference](), [long-context](), [moe](), [nvfp4](), [quantization](), [qwen](), [reasoning](), [technical-how-to](), [tool](), [vllm]() ### AI overview A deployment walkthrough for serving the open-weight Qwen3.8-2.4T-A95B language model on Amazon SageMaker HyperPod with vLLM and NVIDIA B300 GPUs. It covers provisioning, NVFP4 quantization, an OpenAI-compatible endpoint, reasoning, tool calling, and MTP speculative decoding. ### Source excerpt Learn how to deploy Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight model, on Amazon SageMaker HyperPod with vLLM. This walkthrough covers cluster provisioning, NVFP4 quantization, and an OpenAI-compatible endpoint with built-in reasoning, tool calling, and native MTP speculative decoding. ## When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving DevFeed: [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-09T20:31:04Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Optimization](), [model-deployment](), [multimodal](), [Dynamo](), [Inference Performance](), [Embeddings](), [Mixture of Experts (MoE)](), [GPU](), [Open Source](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [computer-vision-video-analytics](), [developer-tools-techniques](), [dynamo](), [embeddings](), [gpu](), [inference](), [low-latency-inference](), [mixture-of-experts-moe](), [model-serving](), [multimodal](), [nvfp4](), [open-source](), [optimization]() ### AI overview This article explains when encode-prefill-decode disaggregation improves multimodal model serving. Using NVIDIA Dynamo, it separates vision encoding from LLM prefill and decode so the stages can batch, schedule, and scale independently. The approach is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, with reported gains of up to 5x in time to first token and 7x in end-to-end response time. ### Source excerpt Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... ## Training Yandex's Alice Omnimodel to Integrate Text and Images DevFeed: [Training Yandex's Alice Omnimodel to Integrate Text and Images]() Original publisher: [Read original article]() Author: formica\_rufa (Яндекс) Published: 2026-09-03T07:03:43Z Content type: tutorial Language: ru Sources: [Яндекс - Как мы делаем Яндекс / Статьи]() Topics: [Large Language Model](), [vlm](), [Artificial Intelligence](), [Mixture of Experts (MoE)](), [qwen](), [Mercury]() Tags: [ai](), [llm](), [ml](), [moe](), [qwen](), [tag-055aee430837](), [tag-61cd5a476b1d](), [tag-831b63de9433](), [tag-86b843454893](), [tag-95a2c958e46b](), [tag-ef0b1bf200df](), [vlm]() ### AI overview Yandex describes its work on an Alice omnimodel that combines a text LLM and a visual VLM into one model for text and image interactions. The article focuses on lessons from training and alignment, including the role of MoE architecture and reinforcement learning. ### Source excerpt Ещё недавно Алиса отвечала на текст и на картинку будто двумя разными голосами. Под капотом и правда жили две генеративные модели: текстовая LLM и визуальная VLM, а между ними -- стена из непрозрачного роутинга, разных форматов ответов и разной вёрстки. Почти год мы сводили их в одну омнимодель -- такую, которая воспринимает текст и изображения как единое целое, без переключений за кадром. Получилось не всё и не сразу, но путь вышел поучительным, и в этой статье я хочу поделиться тем, что мы поняли про обучение таких моделей. Попутно -- несколько неочевидных поворотов: почему за два года до этого та же затея разваливалась, что изменила MoE-архитектура, почему омнипретрейн пришлось собирать с конца и почему один вид RL переезжает на большую модель легко, а другой рассыпается прямо на глазах. Меня зовут Алексей Григорьев, я представляю большую команду разработки омнимодели Яндекса. Вместе с моим коллегой Данилой Кашиным я расскажу про все технические грабли не со стороны наблюдателя, а как их непосредственный собиратель. Но рассказывать я буду с акцентом не на красивом замысле, а на самой болезненной части -- алайнменте. Читать далее ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## Open-weight models are fast on Neon AI Gateway. Here's why DevFeed: [Open-weight models are fast on Neon AI Gateway. Here's why]() Original publisher: [Read original article]() Author: Carlota Soto Published: 2026-08-20T12:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [databricks](), [Inference Performance](), [model-deployment](), [Low-Latency Inference](), [Caching](), [Mixture of Experts (MoE)](), [gpt-oss](), [Large Language Model]() Tags: [caching](), [databricks](), [gpt-oss](), [inference](), [mixture-of-experts-moe](), [optimization](), [performance](), [product](), [speed]() ### AI overview The article explains why open-weight models can deliver low latency on Neon AI Gateway when paired with Databricks Foundation Model APIs and an optimized serving stack. It highlights continuous batching, KV-cache paging, custom kernels, quantization, multi-GPU layouts, prompt caching, and Mixture-of-Experts architectures. ### Source excerpt Open-weight models are built for performance, but how they're served determines whether end users actually feel that speed. Neon AI Gateway calls Databricks Foundation Model APIs for Databricks-hosted open-weight models, a stack that has years of inference engineering behind it. ## Thinking Machines' Inkling: Architecture and Customization Choices DevFeed: [Thinking Machines' Inkling: Architecture and Customization Choices]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-08-18T15:30:36Z Content type: article Language: en Sources: [ByteByteGo]() Topics: [Artificial Intelligence](), [Mixture of Experts (MoE)](), [context window](), [Fine-tuning](), [interfaces](), [data]() Tags: [ai](), [architecture](), [context-window](), [fine-tuning](), [hugging-face](), [mixture-of-experts](), [model](), [real-time]() ### AI overview The article examines the architecture and design choices behind Thinking Machines' Inkling model, including its mixture-of-experts structure, local and global attention, position encoding, multimodal inputs, and adjustable thinking effort. It also notes that Inkling is the company's first model trained from scratch and that its weights are available on Hugging Face under an Apache 2.0 license. ### Source excerpt In this article, we will work through the various choices Thinking Machines made while building Inkling. ## 🗓 This Week In AI Research (1-7 August 26) DevFeed: [🗓 This Week In AI Research (1-7 August 26)]() Original publisher: [Read original article]() Author: Dr. Ashish Bamania Published: 2026-08-13T19:29:25Z Content type: article Language: en Sources: [Into AI]() Topics: [AI Research](), [releases](), [Benchmark](), [Large Language Model](), [Mixture of Experts (MoE)](), [model architecture](), [qwen]() Tags: [ai](), [ai-research](), [algorithm](), [benchmark](), [llms](), [mixture-of-experts](), [model-architecture](), [qwen](), [reasoning](), [releases]() ### AI overview A weekly roundup of AI research and model releases. It highlights Pathway, Bielik AI, and NYU's BDH-CQ reasoning model, which uses in-context learning with recurrent memory and latent-state reasoning, reports ARC-AGI-1 cost-efficiency results, and describes Alibaba's Qwen3.8-Max release and the U-OPSD self-distillation algorithm. ### Source excerpt The top 10 AI research papers and releases that you must know about this week. ## 🗓 This Week In AI Research (25-31 July 26) DevFeed: [🗓 This Week In AI Research (25-31 July 26)]() Original publisher: [Read original article]() Author: Dr. Ashish Bamania Published: 2026-08-07T01:00:43Z Content type: article Language: en Sources: [Into AI]() Topics: [releases](), [Machine learning](), [deepseek](), [foundation-models](), [Mixture of Experts (MoE)](), [lora](), [codex]() Tags: [ai](), [deep-learning](), [deepseek](), [lora](), [ml](), [moe](), [performance](), [releases](), [research]() ### AI overview A weekly roundup of AI research and releases covering DeepSeek-V4-Flash-0731, the Pangram 4 AI-text classification model, the OpenMLE system and its Frontis-MA1-35B agent, and the Metis memory foundation model. ### Source excerpt The top 10 AI research papers and releases this week. ## This Week In AI Research (🗓 9-16 July 26) DevFeed: [This Week In AI Research (🗓 9-16 July 26)]() Original publisher: [Read original article]() Author: Dr. Ashish Bamania Published: 2026-07-22T09:14:26Z Content type: article Language: en Sources: [Into AI]() Topics: [Artificial Intelligence](), [AI Research](), [releases](), [Large Language Model](), [Mixture of Experts (MoE)](), [long-context](), [Open Source Models & Datasets](), [Transformers]() Tags: [ai](), [architecture](), [benchmarks](), [context-window](), [llms](), [models](), [moe](), [open](), [releases](), [research](), [transformers]() ### AI overview A weekly roundup of AI research papers and releases, covering Kimi K3, Expanded Hyper-Connections, VideoChat3, and other developments. It describes Kimi K3's architecture, context window, benchmark performance, and limitations, and summarizes xHC's reported efficiency improvements. ### Source excerpt The top 10 AI research papers and releases this week (Kimi K3, Inkling, WanSong v1.0, Bonsai 27B, and many more) ## Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72 DevFeed: [Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72]() Original publisher: [Read original article]() Author: Kirthi Devleker Published: 2026-07-21T18:30:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Mixture of Experts (MoE)](), [GPU](), [Nvidia](), [AI Platform](), [NCCL](), [networking](), [deepseek]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-platform](), [ai-training](), [collective](), [communication](), [compute](), [data-center-cloud](), [deepseek](), [developer-tools-techniques](), [efficiency](), [featured](), [frontier-model](), [gb300-nvl72](), [gpu](), [infrastructure](), [latency](), [llm-techniques](), [low-latency](), [megatron](), [mixture-of-experts-moe](), [moe](), [nvidia](), [nvl72](), [performance](), [top-stories](), [train](), [training-ai-models]() ### AI overview The article explains how NVIDIA GB300 NVL72 achieved a world record for DeepSeek-V3 671B mixture-of-experts pre-training. It focuses on the communication demands of MoE models, including all-to-all traffic between GPUs, and the need for tightly coupled scale-up and predictable scale-out networking to sustain delivered training performance. ### Source excerpt Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token... ## Laguna S 2.1 is now available on AI Gateway DevFeed: [Laguna S 2.1 is now available on AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-07-21T00:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [AI Models](), [agentic-coding](), [Mixture of Experts (MoE)](), [context window](), [API](), [MLOps](), [AI Research](), [API keys](), [Routing (disambiguation)]() Tags: [agentic-coding](), [ai-models](), [ai-research](), [api-keys](), [context-window](), [cost](), [mixture-of-experts](), [mlops](), [routing](), [sdk](), [support]() ### AI overview Poolside's Laguna S 2.1 is now available through Vercel AI Gateway in free and paid versions, with context windows of 256K and 1M tokens. The open-weight Mixture-of-Experts model supports thinking and no-thinking modes and is designed for agentic coding, long-running tasks, browser tooling, MLOps pipelines, and AI research. ### Source excerpt Laguna S 2.1 from Poolside is now available on AI Gateway. There are 2 versions of the model available: Free version (256K context window): poolside/laguna-s-2.1-free Paid version (1M context window): poolside/laguna-s-2.1 Laguna S 2.1 is an open-weight Mixture-of-Experts model that supports a context window of up to 1M tokens and runs in thinking and no-thinking modes. The model specializes in agentic coding and long-running tasks, including writing and debugging code, running tests, building browser-based tooling, and working on MLOps pipelines and AI research. In thinking mode, Laguna S 2.1 reports 70.2% on Terminal-Bench 2.1, 78.5% on SWE-bench Multilingual, and 59.4% on SWE-Bench Pro. To use Laguna S 2.1, set model to poolside/laguna-s-2.1-free or poolside/laguna-s-2.1 in the AI SDK: AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. Try Laguna S 2.1 in the model playground. Read more ## NVIDIA NVLink: The Scale-Up Network for AI Factories DevFeed: [NVIDIA NVLink: The Scale-Up Network for AI Factories]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-20T15:46:28Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [NVLink](), [AI Factory](), [AI Infrastructure](), [GPU](), [networking](), [Artificial Intelligence](), [Nvidia](), [datacenter](), [Latency](), [Mixture of Experts (MoE)](), [Inference Performance](), [Low Latency]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-factory](), [ai-inference](), [ai-infrastructure](), [collective](), [communication](), [compute](), [data-center](), [featured](), [gpu](), [inference](), [infiniband](), [latency](), [low-latency](), [mixture-of-experts-moe](), [networking](), [nvidia](), [nvlink](), [production](), [scale](), [spectrum-ethernet](), [spectrum-x](), [speed](), [systems](), [vera-rubin]() ### AI overview NVIDIA NVLink is presented as a scale-up networking fabric for AI factories. It provides high-bandwidth, low-latency GPU-to-GPU communication for large AI inference, training, and parallel-computing workloads, with collective-operation acceleration and rack-level resiliency. ### Source excerpt The demand for AI continues to accelerate. Workloads are getting larger, models are becoming more complex, and there is mounting pressure to deploy AI compute... ## Welcome Inkling by Thinking Machines DevFeed: [Welcome Inkling by Thinking Machines]() Original publisher: [Read original article]() Author: ben burtenshaw; merve; Pedro Cuenca; Aritra Roy Gosthipaty; Andres Marafioti Published: 2026-07-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Large Language Model](), [Mixture of Experts (MoE)](), [multimodal](), [hugging face](), [Inference](), [Deployment](), [Fine-tuning](), [Open Source](), [sglang]() Tags: [announcement](), [artificial-intelligence](), [audio](), [context-window](), [deployment](), [fine-tuning](), [generation](), [inference](), [inference-endpoints](), [llama-cpp](), [llms](), [mixture-of-experts](), [mixture-of-experts-moe](), [moe](), [multimodal](), [open-source](), [sglang](), [transformers](), [vllm]() ### AI overview Thinking Machines Lab's Inkling is presented as a large open multimodal language model that accepts image, text, and audio inputs. The article covers its mixture-of-experts architecture, million-token context window, reasoning across modalities, fine-tuning use cases, model variants, and deployment through Hugging Face Inference Endpoints and inference frameworks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading DevFeed: [Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-07-10T18:17:40Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [LLM Techniques](), [Large Language Model](), [GPU](), [Mixture of Experts (MoE)](), [cpu](), [NVLink](), [systems](), [Nvidia]() Tags: [agentic-ai-generative-ai](), [blackwell](), [compute](), [cpu](), [developer-tools-techniques](), [featured](), [gb200](), [gpu](), [grace-cpu](), [large-language-model](), [llm](), [llm-techniques](), [mixture-of-experts-moe](), [nvidia](), [nvl72](), [nvlink](), [simulation-modeling-design](), [training-ai-models](), [vera-rubin](), [work]() ### AI overview This article explains how host offloading in JAX-based large language model training reduces GPU high-bandwidth memory pressure by moving selected activations to pinned host memory and streaming them back during the backward pass. It discusses activation-transfer overlap, NVIDIA Grace Blackwell and GB200 NVL72 systems, and experiments involving Llama 3.1 405B and DeepSeek-V3 671B. ### Source excerpt Large language model (LLM) training workloads increasingly run into GPU memory limits before compute is fully used. Model weights, gradients, optimizer states,... ## A Jargon-Free Learning Hub for AI Foundations DevFeed: [A Jargon-Free Learning Hub for AI Foundations]() Original publisher: [Read original article]() Author: Dr. Ashish Bamania Published: 2026-07-04T11:28:14Z Content type: opinion Language: en Sources: [Dr. Ashish Bamania]() Topics: [Artificial Intelligence](), [Large Language Model](), [Mixture of Experts (MoE)](), [AI Development]() Tags: [ai](), [artificial-intelligence](), [data-science](), [llm](), [machine-learning](), [mixture-of-experts-moe](), [programming](), [publications](), [technology]() ### AI overview The author announces a learning resource focused on high-quality, jargon-free information about Artificial Intelligence. Its planned lessons cover how LLMs generate text, building and training an LLM from scratch, NVIDIA GPUs for AI engineers, and building and training a Mixture-of-Experts LLM from scratch. ### Source excerpt Photo by Google DeepMind on Unsplash 👋🏻 Hey everyone! The internet is full of publications that buzz about every new model release, churn out hot takes on who's winning in AI, and spew doom-and-gloom hype (plus a hundred "10 prompts that will change your life" posts). None of them have made a meaningful difference in my life. Rather, this content is written to provoke emotions and go viral. I have a deep dislike for this type of writing. So, I am fixing this by going the other way and creating a spot where you can find very high-quality, jargon-free information about Artificial Intelligence. Here are four lessons that describe what I'm talking about. A hardware-level tour of how LLMs generate text Build and train an LLM from scratch What every AI engineer must know about NVIDIA GPUs Build and Train a Mixture-of-Experts (MoE) LLM from Scratch If you're interested in really leveling up as an AI engineer, here's your link to join me alongside 10,000+ other smart engineers 👉🏻 Into AI Cheers! ## Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel DevFeed: [Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel]() Original publisher: [Read original article]() Author: Adil Asif; Alexandros Koumparoulis; Wenwen Gao; Sylendran Arunagiri; David Messina; Bernard Nguyen Published: 2026-06-24T16:00:13Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [NeMo](), [Fine-tuning](), [AI Models](), [Mixture of Experts (MoE)](), [distributed-training](), [Generative AI](), [Artificial Intelligence](), [GPU](), [PyTorch](), [vllm](), [sglang]() Tags: [ai](), [ai-models](), [building](), [compute](), [distributed-training](), [fine-tuning](), [framework](), [generative](), [generative-ai](), [gpu](), [infrastructure](), [kernels](), [memory](), [model](), [moe](), [nemo](), [nemotron](), [nvidia](), [open](), [transformers]() ### AI overview This article explains how NVIDIA NeMo AutoModel accelerates fine-tuning of mixture-of-experts models by extending Transformers v5 with Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels. It describes API compatibility, distributed execution, dynamic weight loading, and reported gains of 3.4-3.7x higher training throughput and 29-32% lower GPU memory use. ### Source excerpt NVIDIA NeMo AutoModel is an open library part of the NVIDIA NeMo framework for building custom generative AI models at scale. NeMo AutoModel builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5's dynamic weight loading to bring those optimizations to a broad and growing set of model families. ## Groq on Endless Compute, Inside Claude's Mind, and GLM-5.2 Open Weights - The Tokenizer Edition #32 DevFeed: [Groq on Endless Compute, Inside Claude's Mind, and GLM-5.2 Open Weights - The Tokenizer Edition #32]() Original publisher: [Read original article]() Author: Sairam Sundaresan Published: 2026-06-21T16:45:54Z Content type: article Language: en Sources: [Gradient Ascent]() Topics: [Artificial Intelligence](), [Large Language Model](), [Mixture of Experts (MoE)](), [Inference](), [multimodal](), [groq](), [Claude]() Tags: [ai](), [ai-ml](), [github](), [inference](), [llm](), [mixture-of-experts](), [multimodal]() ### AI overview The Tokenizer Edition #32 curates AI and machine learning resources covering model interpretability, compute demand, open-weight models, multimodal video processing, speculative decoding, reinforcement learning, agent testing, and tools for cheaper or safer inference. ### Source excerpt This week's most valuable AI resources ## Nemotron 3 Ultra now available on AI Gateway DevFeed: [Nemotron 3 Ultra now available on AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-06-04T07:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [Nemotron](), [Mixture of Experts (MoE)](), [API](), [context window](), [Nvidia](), [Vercel](), [Inference](), [SDKs](), [vercel ai sdk](), [Latency]() Tags: [ai](), [api](), [context-window](), [inference](), [latency](), [mixture-of-experts](), [nemotron](), [nvidia](), [performance](), [sdk](), [tokens](), [tool](), [vercel]() ### AI overview Nemotron 3 Ultra from Nvidia is now available through Vercel AI Gateway. The open Mixture-of-Experts reasoning model supports long-running, multi-turn agent workflows with a 1M token context window, tool use, delegation, and error recovery. The gateway provides model access, usage and cost tracking, retries, failover, provider sorting, and performance optimizations. ### Source excerpt Nemotron 3 Ultra from Nvidia is now available on Vercel AI Gateway. Nemotron 3 Ultra is an open Mixture-of-Experts reasoning model built for orchestrating long-running agent workflows, with a 1M token context window. The model targets multi-turn agent workflows: planning, tool use, sub-agent delegation, and error recovery. Throughput reaches up to 350 tokens per second, with up to 30% lower cost on agentic tasks. To use Nemotron 3 Ultra, set model to nvidia/nemotron-3-ultra-550b-a55b in the AI SDK. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, dynamic provider sorting by latency and cost, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. Learn more about AI Gateway, view the AI Gateway model leaderboard or try it in our model playground. Read more ## Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains DevFeed: [Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains]() Original publisher: [Read original article]() Author: Nikita Pavlichenko Published: 2026-06-01T15:45:17Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Mixture of Experts (MoE)](), [Low-Latency Inference](), [Inference](), [Retrieval Augmented Generation (RAG)](), [Routing (disambiguation)](), [Benchmark](), [code-completion](), [hugging face]() Tags: [ai](), [benchmark](), [code](), [code-generation](), [coding](), [inference](), [latency](), [low-latency-inference](), [mixture-of-experts](), [model](), [rag](), [routing](), [software]() ### AI overview JetBrains introduces Mellum2, a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. It activates 2.5B parameters per token to provide efficient, low-latency inference for routing, retrieval-augmented generation, summarization, sub-agent tasks, and high-throughput coding features. The model is released under the Apache 2.0 license and is reported to deliver competitive benchmark performance with more than twice the inference speed of similarly sized models. ### Source excerpt - Mellum2 is a 12B-parameter Mixture-of-Experts model trained from scratch on natural language and code. - The model activates only 2.5B parameters per token, making it efficient for high-throughput, low-latency inference. Mellum2 is can be used for routing, RAG, summarization, sub-agents, high-throughput coding features, and private deployments. - It is released under the Apache 2.0 license. ## Optimizing Storage for Large Model Deployments on Inference Clouds DevFeed: [Optimizing Storage for Large Model Deployments on Inference Clouds]() Original publisher: [Read original article]() Author: Brett Snyder Published: 2026-04-21T20:10:14Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Inference](), [model-deployment](), [GPU](), [Mixture of Experts (MoE)](), [cloud-infrastructure](), [Cloud](), [Latency](), [Artificial Intelligence]() Tags: [agentic](), [ai](), [architecture](), [cloud](), [cost](), [data](), [data-gravity](), [engineering](), [gpu](), [inference](), [infrastructure](), [latency](), [model]() ### AI overview This article explains how high-bandwidth storage can reduce model-loading delays for large deployments on inference clouds. It argues that moving very large model weights into VRAM can create cold starts, idle GPU capacity, and delays in agentic workflows, making storage optimized for GPU consumption increasingly important. ### Source excerpt We have moved past the point where a 70GB model was considered "heavy." With the rise of models like DeepSeek-V3, the GLM series, and other massive Mixture-of-Experts (MoE) architectures, the industry is now grappling with weights exceeding 700GB in optimized formats--and well over 1.2TB in full precision. And parameters keep climbing--Epoch's AI data tracks frontier models now reaching into the trillions of parameters, with no sign of plateau. At this scale, "Data Gravity" isn't just a metaphor; it is a structural bottleneck. If your storage architecture isn't optimized for these massive assets, the latency of moving weights into VRAM can undermine the unit economics of your entire GPU fleet. Every time an agent orchestrating a multi-step workflow hands off to a different specialized model, the user on the other end is waiting--and what they're waiting on is your storage layer, not your intelligence. Deploying production workloads to an inference cloud that provides both GPUs and storage optimized for GPU consumption will often be non-negotiable as model sizes continue to grow. The Cost of the "Idle Wait" When deploying GPU infrastructure, the most expensive resource can be idle silicon. A standard 1Gbps connection is fundamentally incapable of supporting modern large-scale models, requiring hours to "pull" a single checkpoint. Even at 10Gbps, the "Data Tax"--the time spent waiting for weights to load--can lead to 15-20 minute cold starts. In agentic workflows, where a primary agent may need to spin up specialized "expert" nodes on demand, these delays can create a cascading failure. If your infrastructure can't scale a node and load its model in under two minutes, real-time agentic behavior becomes impossible. This might look like a coding agent calling a specialized security-auditing model mid-task, where a five-minute cold start means the user has already abandoned the session. The ROI of High-Bandwidth Storage To understand why high-throughput storage matters, we ha [Next page]()