# moe

Published articles for moe.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each

DevFeed: [Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each](<https://devfeed.tech/articles/dense-vs-moe-models-active-parameters-throughput-and-when-to-choose-each-26912.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/dense-vs-moe-models-active-parameters-throughput-and-when-to-choose-each/>)

Author: Elizabeth Goodman

Published: 2026-09-15T17:00:11Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nemo](<https://devfeed.tech/tags/nemo.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [performance](<https://devfeed.tech/tags/performance.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

This article explains how dense and Mixture-of-Experts models activate parameters, compares their effects on throughput, memory cost, and serving complexity, and discusses when each architecture fits different deployment constraints. It uses Nemotron 3.5 Lightning as an example of an MoE model.

### Source excerpt

How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the...

## Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine

DevFeed: [Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine](<https://devfeed.tech/articles/accelerating-dropless-moe-training-in-jax-with-nvidia-transformer-engine-21079.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/accelerating-dropless-moe-training-in-jax-with-nvidia-transformer-engine/>)

Author: Tanya Lenz

Published: 2026-09-14T16:39:15Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>), [GB200](<https://devfeed.tech/topics/gb200.md>), [Network](<https://devfeed.tech/topics/network.md>), [Python](<https://devfeed.tech/topics/python.md>), [deepseek](<https://devfeed.tech/topics/deepseek.md>), [mixtral](<https://devfeed.tech/topics/mixtral.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [communication](<https://devfeed.tech/tags/communication.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [gb200](<https://devfeed.tech/tags/gb200.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [mixtral](<https://devfeed.tech/tags/mixtral.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [moe](<https://devfeed.tech/tags/moe.md>), [networks](<https://devfeed.tech/tags/networks.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [performance](<https://devfeed.tech/tags/performance.md>), [python](<https://devfeed.tech/tags/python.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [tensors](<https://devfeed.tech/tags/tensors.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training-ai-models](<https://devfeed.tech/tags/training-ai-models.md>)

### AI overview

This article explains how NVIDIA Transformer Engine and JAX kernel optimizations accelerate dropless Mixture of Experts (MoE) training. It describes bottlenecks from token routing, expert dispatch and gathering, all-to-all communication, and ragged expert matrix operations. In DeepSeek-V3 training on NVIDIA GB200, the optimized approach increased performance from 103 to 1,068 TFLOPS per GPU, a 10.4x improvement.

### Source excerpt

Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE...

## Открываем претрейн Alice AI Search: как устроена модель быстрых ответов Алисы на Поиске

DevFeed: [Открываем претрейн Alice AI Search: как устроена модель быстрых ответов Алисы на Поиске](<https://devfeed.tech/articles/alice-ai-search-24897.md>)

Original publisher: [Read original article](<https://habr.com/ru/companies/yandex/articles/1080654/>)

Author: pet67 (Яндекс)

Published: 2026-09-11T06:05:13Z

Content type: article

Language: ru

Sources: [Яндекс - Как мы делаем Яндекс / Статьи](<https://devfeed.tech/sources/source.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI search](<https://devfeed.tech/topics/ai-search.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [alice-ai](<https://devfeed.tech/tags/alice-ai.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [llm](<https://devfeed.tech/tags/llm.md>), [ml](<https://devfeed.tech/tags/ml.md>), [moe](<https://devfeed.tech/tags/moe.md>), [rl](<https://devfeed.tech/tags/rl.md>), [tag-178bc8f01f24](<https://devfeed.tech/tags/tag-178bc8f01f24.md>), [tag-4004cf5948d3](<https://devfeed.tech/tags/tag-4004cf5948d3.md>), [tag-61cd5a476b1d](<https://devfeed.tech/tags/tag-61cd5a476b1d.md>), [tag-d89cae10e887](<https://devfeed.tech/tags/tag-d89cae10e887.md>), [tag-e6d9cc1f0757](<https://devfeed.tech/tags/tag-e6d9cc1f0757.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

This developer article explains the Alice AI Search pipeline for generating fast answers, including its search and context-processing stages, shorter information contexts, a sparse Mixture-of-Experts architecture combined with an Encoder-Decoder, and online reinforcement learning from user behavior signals. It also announces the open release of the Alice AI-T5-35B-A0.6B Base model, with external inference available through Hugging Face Transformers while optimized production inference remains internal to Yandex.

### Source excerpt

Быстрый ответ Алисы AI -- это самый массовый генеративный продукт Яндекса и первое соприкосновение с Алисой для пользователей Поиска. Даже в час пиковой нагрузки пользователь должен получить лаконичный ответ за считаные секунды. Для этого мы, команда Alice AI Search, адаптируем весь пайплайн быстрых ответов -- от собственного претрейна с кастомной архитектурой до онлайн-rl-обучения на поведенческие сигналы пользователей. В статье разберём, как устроен генеративный ответ в Поиске, и расскажем про основные улучшения июньского релиза: как мы ускорили ответы за счёт коротких инфоконтекстов, зачем совместили Encoder-Decoder с разреженной MoE-архитектурой и как обучение на реальных пользовательских сигналах повлияло на качество и использование продукта. Кроме того, мы выложили в открытый доступ обученную с нуля модель Alice AI-T5-35B-A0.6B Base с тем ограничением, что внешним пользователям доступен инференс через Hugging Face Transformers, а оптимизированный production-инференс пока доступен только внутри Яндекса. Читать далее

## Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

DevFeed: [Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM](<https://devfeed.tech/articles/deploying-qwen3-8-2-4t-a95b-on-amazon-sagemaker-hyperpod-with-vllm-4731.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/deploying-qwen3-8-2-4t-a95b-on-amazon-sagemaker-hyperpod-with-vllm/>)

Author: Dmitry Soldatkin

Published: 2026-09-09T22:26:29Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Deployment](<https://devfeed.tech/topics/deployment.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-hyperpod](<https://devfeed.tech/tags/amazon-sagemaker-hyperpod.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [inference](<https://devfeed.tech/tags/inference.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [tool](<https://devfeed.tech/tags/tool.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

A deployment walkthrough for serving the open-weight Qwen3.8-2.4T-A95B language model on Amazon SageMaker HyperPod with vLLM and NVIDIA B300 GPUs. It covers provisioning, NVFP4 quantization, an OpenAI-compatible endpoint, reasoning, tool calling, and MTP speculative decoding.

### Source excerpt

Learn how to deploy Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight model, on Amazon SageMaker HyperPod with vLLM. This walkthrough covers cluster provisioning, NVFP4 quantization, and an OpenAI-compatible endpoint with built-in reasoning, tool calling, and native MTP speculative decoding.

## Training Yandex's Alice Omnimodel to Integrate Text and Images

DevFeed: [Training Yandex's Alice Omnimodel to Integrate Text and Images](<https://devfeed.tech/articles/ai-vlm-llm-24891.md>)

Original publisher: [Read original article](<https://habr.com/ru/companies/yandex/articles/1077274/>)

Author: formica\_rufa (Яндекс)

Published: 2026-09-03T07:03:43Z

Content type: tutorial

Language: ru

Sources: [Яндекс - Как мы делаем Яндекс / Статьи](<https://devfeed.tech/sources/source.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [Mercury](<https://devfeed.tech/topics/mercury-lang.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [llm](<https://devfeed.tech/tags/llm.md>), [ml](<https://devfeed.tech/tags/ml.md>), [moe](<https://devfeed.tech/tags/moe.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [tag-055aee430837](<https://devfeed.tech/tags/tag-055aee430837.md>), [tag-61cd5a476b1d](<https://devfeed.tech/tags/tag-61cd5a476b1d.md>), [tag-831b63de9433](<https://devfeed.tech/tags/tag-831b63de9433.md>), [tag-86b843454893](<https://devfeed.tech/tags/tag-86b843454893.md>), [tag-95a2c958e46b](<https://devfeed.tech/tags/tag-95a2c958e46b.md>), [tag-ef0b1bf200df](<https://devfeed.tech/tags/tag-ef0b1bf200df.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

Yandex describes its work on an Alice omnimodel that combines a text LLM and a visual VLM into one model for text and image interactions. The article focuses on lessons from training and alignment, including the role of MoE architecture and reinforcement learning.

### Source excerpt

Ещё недавно Алиса отвечала на текст и на картинку будто двумя разными голосами. Под капотом и правда жили две генеративные модели: текстовая LLM и визуальная VLM, а между ними -- стена из непрозрачного роутинга, разных форматов ответов и разной вёрстки. Почти год мы сводили их в одну омнимодель -- такую, которая воспринимает текст и изображения как единое целое, без переключений за кадром. Получилось не всё и не сразу, но путь вышел поучительным, и в этой статье я хочу поделиться тем, что мы поняли про обучение таких моделей. Попутно -- несколько неочевидных поворотов: почему за два года до этого та же затея разваливалась, что изменила MoE-архитектура, почему омнипретрейн пришлось собирать с конца и почему один вид RL переезжает на большую модель легко, а другой рассыпается прямо на глазах. Меня зовут Алексей Григорьев, я представляю большую команду разработки омнимодели Яндекса. Вместе с моим коллегой Данилой Кашиным я расскажу про все технические грабли не со стороны наблюдателя, а как их непосредственный собиратель. Но рассказывать я буду с акцентом не на красивом замысле, а на самой болезненной части -- алайнменте. Читать далее

## Hot Chips 2026: Applying High Bandwidth Flash (HBF)

DevFeed: [Hot Chips 2026: Applying High Bandwidth Flash (HBF)](<https://devfeed.tech/articles/hot-chips-2026-applying-high-bandwidth-flash-hbf-13990.md>)

Original publisher: [Read original article](<https://chipsandcheese.com/p/hot-chips-2026-applying-high-bandwidth>)

Author: Chester Lam

Published: 2026-08-23T22:51:05Z

Content type: article

Language: en

Sources: [Chips and Cheese](<https://devfeed.tech/sources/chips-and-cheese.md>)

Topics: [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [moe](<https://devfeed.tech/topics/moe.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [dram](<https://devfeed.tech/tags/dram.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [moe](<https://devfeed.tech/tags/moe.md>), [ssd](<https://devfeed.tech/tags/ssd.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This article examines how High Bandwidth Flash (HBF) could support machine learning workloads. HBF does not yet have products; the discussion uses simulations and projections to explore software strategies, including moving Mixture-of-Experts components or KV cache data between HBF and faster memory, with vLLM as an example.

### Source excerpt

Machine learning workloads have an insatiable appetite for DRAM capacity. Flash memory is cheaper per gigabyte of capacity than DRAM. Could it offer a way out?

## How Much Memory Does Your Agent Actually Need?

DevFeed: [How Much Memory Does Your Agent Actually Need?](<https://devfeed.tech/articles/how-much-memory-does-your-agent-actually-need-7262.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ibm-research/altk-evolve-hmm>)

Author: Vatche Isahagian; Gaodan Fang; Jayaram Radhakrishnan; Punleuk Oum; Ashwath Vaithinathan Aravindan; Evelyn Duesterwald; G Thomas; Vinod Muthusamy; Merve Unuvar; Ayhan Sebin

Published: 2026-08-18T18:09:38Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>), [deepseek](<https://devfeed.tech/topics/deepseek.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [caching](<https://devfeed.tech/tags/caching.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gpt-oss](<https://devfeed.tech/tags/gpt-oss.md>), [inference](<https://devfeed.tech/tags/inference.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [performance](<https://devfeed.tech/tags/performance.md>), [production](<https://devfeed.tech/tags/production.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

ALTK-Evolve equips agents with agentic memory by distilling reusable guidelines from past trajectories and injecting them at inference time without weight updates or human annotation. An evaluation across eight models found that the effective amount of memory varies by model: stronger models benefited from full guideline sets, weaker models from compact cores plus task-specific retrieval, and saturated models showed no measurable improvement. For gpt-oss-120b, selective retrieval improved task completion by 16.1 percentage points with only a 5% token increase.

### Source excerpt

Equipping an agent with agentic memory sounds simple: distill lessons from its past work, put them back in context, and more experience should mean better performance. It doesn't always work that way. When we scaled the evaluation to eight models -- from a 30B dense model to frontier proprietary systems -- one finding stood out: Agentic memory is not a feature you switch on. It's a dose you calibrate to the model.

## 🗓 This Week In AI Research (25-31 July 26)

DevFeed: [🗓 This Week In AI Research (25-31 July 26)](<https://devfeed.tech/articles/this-week-in-ai-research-25-31-july-26-18286.md>)

Original publisher: [Read original article](<https://www.intoai.pub/p/this-week-in-ai-research-25-31-july>)

Author: Dr. Ashish Bamania

Published: 2026-08-07T01:00:43Z

Content type: article

Language: en

Sources: [Into AI](<https://devfeed.tech/sources/into-ai.md>)

Topics: [releases](<https://devfeed.tech/topics/releases.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [deepseek](<https://devfeed.tech/topics/deepseek.md>), [foundation-models](<https://devfeed.tech/topics/foundation-models.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [lora](<https://devfeed.tech/topics/lora.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [lora](<https://devfeed.tech/tags/lora.md>), [ml](<https://devfeed.tech/tags/ml.md>), [moe](<https://devfeed.tech/tags/moe.md>), [performance](<https://devfeed.tech/tags/performance.md>), [releases](<https://devfeed.tech/tags/releases.md>), [research](<https://devfeed.tech/tags/research.md>)

### AI overview

A weekly roundup of AI research and releases covering DeepSeek-V4-Flash-0731, the Pangram 4 AI-text classification model, the OpenMLE system and its Frontis-MA1-35B agent, and the Metis memory foundation model.

### Source excerpt

The top 10 AI research papers and releases this week.

## Ling 3.0 Tiny is now available on AI Gateway

DevFeed: [Ling 3.0 Tiny is now available on AI Gateway](<https://devfeed.tech/articles/ling-3-0-tiny-is-now-available-on-ai-gateway-1001.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/ling-3-0-tiny-is-now-available-on-ai-gateway>)

Author: Jerilyn Zheng

Published: 2026-08-06T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [moe](<https://devfeed.tech/topics/moe.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [SDKs](<https://devfeed.tech/topics/sdks.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [caching](<https://devfeed.tech/tags/caching.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [sdk](<https://devfeed.tech/tags/sdk.md>)

### AI overview

Ling 3.0 Tiny from ANT Group is available on Vercel AI Gateway, initially in a free slot. It is a Mixture-of-Experts model with 7.9B total parameters, approximately 1.3B active parameters per token, a 256K-token context window, and support for up to 32K output tokens. The article also explains how to use it through the AI SDK, the model playground, and coding agents.

### Source excerpt

Ling 3.0 Tiny from ANT Group is now on AI Gateway, free to use till 8:00am PT on 8/14. Ling 3.0 Tiny takes the free slot from Ling 3.0 Flash. Ling 3.0 Tiny is a MOE model with 7.9B total parameters and about 1.3B active per token, a 256K token context window, and up to 32K output tokens. The model is built for responsive agents, instruction following, and multi-turn conversation, with native function calling and prompt caching. To use Ling 3.0 Tiny, set model to inclusionai/ling-3.0-tiny-free in the AI SDK. On August 14th, the new model name will be inclusionai/ling-3.0-tiny. Try Ling 3.0 Tiny in the model playground. To use it in a coding agent, run vercel ai-gateway coding-agents setup and select inclusionai/ling-3.0-tiny-free inside the agent. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. Read more

## This Week In AI Research (🗓 9-16 July 26)

DevFeed: [This Week In AI Research (🗓 9-16 July 26)](<https://devfeed.tech/articles/this-week-in-ai-research-9-16-july-26-18287.md>)

Original publisher: [Read original article](<https://www.intoai.pub/p/this-week-in-ai-research-9-16-july>)

Author: Dr. Ashish Bamania

Published: 2026-07-22T09:14:26Z

Content type: article

Language: en

Sources: [Into AI](<https://devfeed.tech/sources/into-ai.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>), [releases](<https://devfeed.tech/topics/releases.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [llms](<https://devfeed.tech/tags/llms.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [open](<https://devfeed.tech/tags/open.md>), [releases](<https://devfeed.tech/tags/releases.md>), [research](<https://devfeed.tech/tags/research.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

A weekly roundup of AI research papers and releases, covering Kimi K3, Expanded Hyper-Connections, VideoChat3, and other developments. It describes Kimi K3's architecture, context window, benchmark performance, and limitations, and summarizes xHC's reported efficiency improvements.

### Source excerpt

The top 10 AI research papers and releases this week (Kimi K3, Inkling, WanSong v1.0, Bonsai 27B, and many more)

## Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72

DevFeed: [Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72](<https://devfeed.tech/articles/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-6939.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/>)

Author: Kirthi Devleker

Published: 2026-07-21T18:30:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [AI Platform](<https://devfeed.tech/topics/ai-platform.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [networking](<https://devfeed.tech/topics/networking.md>), [deepseek](<https://devfeed.tech/topics/deepseek.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [featured](<https://devfeed.tech/tags/featured.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [gb300-nvl72](<https://devfeed.tech/tags/gb300-nvl72.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-techniques](<https://devfeed.tech/tags/llm-techniques.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [megatron](<https://devfeed.tech/tags/megatron.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [performance](<https://devfeed.tech/tags/performance.md>), [top-stories](<https://devfeed.tech/tags/top-stories.md>), [train](<https://devfeed.tech/tags/train.md>), [training-ai-models](<https://devfeed.tech/tags/training-ai-models.md>)

### AI overview

The article explains how NVIDIA GB300 NVL72 achieved a world record for DeepSeek-V3 671B mixture-of-experts pre-training. It focuses on the communication demands of MoE models, including all-to-all traffic between GPUs, and the need for tightly coupled scale-up and predictable scale-out networking to sustain delivered training performance.

### Source excerpt

Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token...

## Welcome Inkling by Thinking Machines

DevFeed: [Welcome Inkling by Thinking Machines](<https://devfeed.tech/articles/welcome-inkling-by-thinking-machines-7502.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/thinkingmachines-inkling>)

Author: ben burtenshaw; merve; Pedro Cuenca; Aritra Roy Gosthipaty; Andres Marafioti

Published: 2026-07-15T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [multimodal](<https://devfeed.tech/topics/multimodal.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [sglang](<https://devfeed.tech/topics/sglang.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [audio](<https://devfeed.tech/tags/audio.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [generation](<https://devfeed.tech/tags/generation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

Thinking Machines Lab's Inkling is presented as a large open multimodal language model that accepts image, text, and audio inputs. The article covers its mixture-of-experts architecture, million-token context window, reasoning across modalities, fine-tuning use cases, model variants, and deployment through Hugging Face Inference Endpoints and inference frameworks.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

DevFeed: [Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel](<https://devfeed.tech/articles/accelerating-transformers-fine-tuning-with-nvidia-nemo-automodel-7374.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel>)

Author: Adil Asif; Alexandros Koumparoulis; Wenwen Gao; Sylendran Arunagiri; David Messina; Bernard Nguyen

Published: 2026-06-24T16:00:13Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [NeMo](<https://devfeed.tech/topics/nemo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [sglang](<https://devfeed.tech/topics/sglang.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [building](<https://devfeed.tech/tags/building.md>), [compute](<https://devfeed.tech/tags/compute.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nemo](<https://devfeed.tech/tags/nemo.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open](<https://devfeed.tech/tags/open.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

This article explains how NVIDIA NeMo AutoModel accelerates fine-tuning of mixture-of-experts models by extending Transformers v5 with Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels. It describes API compatibility, distributed execution, dynamic weight loading, and reported gains of 3.4-3.7x higher training throughput and 29-32% lower GPU memory use.

### Source excerpt

NVIDIA NeMo AutoModel is an open library part of the NVIDIA NeMo framework for building custom generative AI models at scale. NeMo AutoModel builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5's dynamic weight loading to bring those optimizations to a broad and growing set of model families.

## DeepSeek-V4: a million-token context that agents can actually use

DevFeed: [DeepSeek-V4: a million-token context that agents can actually use](<https://devfeed.tech/articles/deepseek-v4-a-million-token-context-that-agents-can-actually-use-7162.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/deepseekv4>)

Author: ben burtenshaw

Published: 2026-04-24T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [deepseek](<https://devfeed.tech/topics/deepseek.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cache](<https://devfeed.tech/tags/cache.md>), [community](<https://devfeed.tech/tags/community.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [moe](<https://devfeed.tech/tags/moe.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [research](<https://devfeed.tech/tags/research.md>)

### AI overview

DeepSeek-V4 is presented as an open model designed for long-running agentic workloads. The article explains how its architecture reduces the inference FLOPs and KV cache memory required for million-token contexts, and describes post-training choices intended to improve extended tool-use trajectories. It highlights Compressed Sparse Attention and a lightning indexer as mechanisms behind the efficiency gains, with comparisons against DeepSeek-V3.2 and grouped-query attention.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Gemma 4 on AI Gateway

DevFeed: [Gemma 4 on AI Gateway](<https://devfeed.tech/articles/gemma-4-on-ai-gateway-948.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/gemma-4-on-ai-gateway>)

Author: Jerilyn Zheng

Published: 2026-04-02T07:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [cost](<https://devfeed.tech/tags/cost.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [google](<https://devfeed.tech/tags/google.md>), [inference](<https://devfeed.tech/tags/inference.md>), [json](<https://devfeed.tech/tags/json.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [observability](<https://devfeed.tech/tags/observability.md>), [routing](<https://devfeed.tech/tags/routing.md>), [vercel](<https://devfeed.tech/tags/vercel.md>), [vision](<https://devfeed.tech/tags/vision.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Vercel AI Gateway now offers Google's Gemma 4 26B MoE and 31B Dense models. The release highlights model capabilities, tradeoffs between latency and output quality, and gateway features for routing, retries, usage, cost, and observability.

### Source excerpt

Gemma 4 26B (MoE) and 31B (Dense) from Google are now available on Vercel AI Gateway. Built on the same architecture as Gemini 3, both open models support function-calling, agentic workflows, structured JSON output, and system instructions. Both support up to 256K context, 140+ languages, and native vision. 26B (MoE): Activates only 3.8B of its 26B total parameters during inference, optimized for lower latency and faster tokens-per-second. 31B (Dense): All parameters are active during inference, targeting higher output quality. Better suited as a foundation for fine-tuning. To use Gemma 4, set model to google/gemma-4-31b-it or google/gemma-4-26b-a4b-it in the AI SDK. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, observability, Bring Your Own Key support, and intelligent provider routing with automatic retries. Learn more about AI Gateway, view the AI Gateway model leaderboard or try it in our model playground. Read more

## Welcome Gemma 4: Frontier multimodal intelligence on device

DevFeed: [Welcome Gemma 4: Frontier multimodal intelligence on device](<https://devfeed.tech/articles/welcome-gemma-4-frontier-multimodal-intelligence-on-device-7214.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/gemma4>)

Author: merve; Pedro Cuenca; Sergio Paniego; ben burtenshaw; Steven Zheng; Alvaro Bartolome; Nathan Habib

Published: 2026-04-02T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [gemma4](<https://devfeed.tech/topics/gemma4.md>), [multimodal](<https://devfeed.tech/topics/multimodal.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [MLX](<https://devfeed.tech/topics/mlx.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [webgpu](<https://devfeed.tech/topics/webgpu.md>)

Tags: [audio](<https://devfeed.tech/tags/audio.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma4](<https://devfeed.tech/tags/gemma4.md>), [generation](<https://devfeed.tech/tags/generation.md>), [google](<https://devfeed.tech/tags/google.md>), [guide](<https://devfeed.tech/tags/guide.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [local](<https://devfeed.tech/tags/local.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [mlx](<https://devfeed.tech/tags/mlx.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [open](<https://devfeed.tech/tags/open.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [science](<https://devfeed.tech/tags/science.md>), [speed](<https://devfeed.tech/tags/speed.md>)

### AI overview

Gemma 4 is presented as an open, Apache 2-licensed family of multimodal models supporting image, video, text, and, for selected variants, audio inputs. The article highlights on-device deployment, long-context support, compatibility across libraries and devices, quantization, benchmark results, and implementation options including llama.cpp, MLX, WebGPU, and Rust.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## LLM Inference Benchmarking - Measure What Matters

DevFeed: [LLM Inference Benchmarking - Measure What Matters](<https://devfeed.tech/articles/llm-inference-benchmarking-measure-what-matters-19901.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/llm-inference-benchmarking>)

Author: Rithish Ramesh

Published: 2026-02-06T14:46:06Z

Content type: article

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compute](<https://devfeed.tech/tags/compute.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [performance](<https://devfeed.tech/tags/performance.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

This article examines LLM inference benchmarking as a complex systems problem spanning hardware and software. It focuses on how latency, throughput, concurrency, and cost interact, and distinguishes the compute-bound prefill phase from the memory-bound decode phase.

### Source excerpt

Production-grade LLM inference is a complex systems challenge, requiring deep co-designs - from hardware primitives (FLOPs, memory bandwidth, and interconnects) to sophisticated software layers - across the entire stack. Given the hardware variability across GPU providers like NVIDIA and AMD - including generational differences in numeric type performance (FP8, BF16, NVFP4 etc), HBM bandwidth and capacity, peak FLOPs etc - optimal performance is never guaranteed. It depends on the software's ability to maximize FLOPs utilization during prefill, maximize bandwidth efficiency during decode, optimize expert routing in MoE models, discover optimal parallelism strategies, and more. As inference hardware costs remain high, squeezing maximum performance to improve unit economics is a primary objective for AI teams. We are currently in an era of intense hardware-software co-design that will redefine performance and cost efficiency. Consequently, benchmarking must evolve to track three critical pillars: end-to-end model performance, micro-benchmarking of isolated components and a structured way to go after performance improvements. This article focuses on the LLM performance domain and analyzes the interplay between latency, throughput, concurrency, and cost. Prefill and Decode: The two phases of Inference LLM Inference works in two-phases: prefill and decode. The prefill phase is where the entire input goes through the model's forward pass which includes self-attention, add & norm, and pass through the hidden layers of the model's feed forward network. This phase is extremely compute bound. FLOPs per byte transferred (arithmetic intensity) for the prefill phase is very high. In simpler terms, the GPU is spending more time computing than waiting for the data from memory. On the other hand, the decode phase is memory bound. For every token that is generated, decode needs to load the entire weight matrix, KV cache from the HBM, generate one token, and write it back to the HBM

## NVIDIA Nemotron Nano v2: Long-Context Reasoning and Efficient Inference in Smaller Models

DevFeed: [NVIDIA Nemotron Nano v2: Long-Context Reasoning and Efficient Inference in Smaller Models](<https://devfeed.tech/articles/the-future-of-agentic-ai-is-small-35023.md>)

Original publisher: [Read original article](<https://read.theaimerge.com/p/the-future-of-agentic-ai-is-small>)

Author: Alex Razvant

Published: 2025-11-15T14:47:33Z

Content type: article

Language: en

Sources: [Neural Bits](<https://devfeed.tech/sources/neural-bits.md>)

Topics: [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [inference](<https://devfeed.tech/tags/inference.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [model-architecture](<https://devfeed.tech/tags/model-architecture.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

This technical article examines NVIDIA's Nemotron Nano v2 Transformer-Hybrid models, focusing on their long-context reasoning, fast inference, and reported performance against larger open models. It also describes the broader Nemotron family's open models, datasets, and fine-tuning recipes for agentic AI systems.

### Source excerpt

How NVIDIA's Nemotron Nano V2 SLM, built for long-context reasoning, fast inference, can compete with models 4-5x its size.

## Welcome GPT OSS, the new open-source model family from OpenAI!

DevFeed: [Welcome GPT OSS, the new open-source model family from OpenAI!](<https://devfeed.tech/articles/welcome-gpt-oss-the-new-open-source-model-family-from-openai-7567.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/welcome-openai-gpt-oss>)

Author: Vaibhav Srivastav; Pedro Cuenca; Lewis Tunstall; Clem 🤗; Matthew Carrigan; Clémentine Fourrier; Célina Hanouti; Lucain Pouget; Marc Sun; Simon Pagezy

Published: 2025-08-05T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [inference-providers](<https://devfeed.tech/topics/inference-providers.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [moe](<https://devfeed.tech/topics/moe.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>), [Python](<https://devfeed.tech/topics/python.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [community](<https://devfeed.tech/tags/community.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-oss](<https://devfeed.tech/tags/gpt-oss.md>), [inference-providers](<https://devfeed.tech/tags/inference-providers.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [llm](<https://devfeed.tech/tags/llm.md>), [moe](<https://devfeed.tech/tags/moe.md>), [ollama](<https://devfeed.tech/tags/ollama.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openai](<https://devfeed.tech/tags/openai.md>), [python](<https://devfeed.tech/tags/python.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

Hugging Face welcomes OpenAI's gpt-oss open-source model family. The article describes the models' Apache 2.0 licensing, local deployment options, reasoning and tool-use capabilities, MoE architecture, quantization, supported inference implementations, and access through Inference Providers and the Responses API.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## DeepSeek's reported efficiency and Nvidia's stock decline

DevFeed: [DeepSeek's reported efficiency and Nvidia's stock decline](<https://devfeed.tech/articles/nvidia-deepseek-ai-24007.md>)

Original publisher: [Read original article](<https://habr.com/ru/companies/redmadrobot/articles/877416/>)

Author: redmadrobot (red\_mad\_robot)

Published: 2025-01-28T16:53:47Z

Content type: opinion

Language: ru

Sources: [Redmadrobot EN](<https://devfeed.tech/sources/redmadrobot-en.md>), [Redmadrobot RU](<https://devfeed.tech/sources/redmadrobot-ru.md>)

Topics: [deepseek](<https://devfeed.tech/topics/deepseek.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [moe](<https://devfeed.tech/topics/moe.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [ChatGPT](<https://devfeed.tech/topics/chatgpt.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [openai](<https://devfeed.tech/tags/openai.md>), [tag-13e2af703774](<https://devfeed.tech/tags/tag-13e2af703774.md>), [tag-61cd5a476b1d](<https://devfeed.tech/tags/tag-61cd5a476b1d.md>)

### AI overview

The article argues that DeepSeek's reported lower training and inference costs contributed to concern about Nvidia's GPU-demand business model and coincided with a sharp Nvidia stock decline. It attributes the claimed efficiency to reinforcement learning with GRPO and a Mixture of Experts architecture.

### Source excerpt

Акции NVIDIA рухнули, потеряв за одну ночь $600 миллиардов рыночной капитализации -- крупнейший дневной обвал на рынке ценных бумаг в истории США. Это событие стало шоком для рынка GPU, будущее которого, как казалось, предопределено на годы вперед. Причиной стал запуск китайской нейросети DeepSeek, которая не только превзошла западных конкурентов, таких как ChatGPT, но и стоила в разработке в тысячу раз дешевле. Этот прорыв поставил под угрозу бизнес-модель NVIDIA, основанную на стабильном спросе на их GPU. Разбираемся, как так вышло, вместе с Валерой Ковальским, CEO NDT by red_mad_robot. Читать далее