# LLM Techniques

Published articles for LLM Techniques.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72

DevFeed: [Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72](<https://devfeed.tech/articles/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-6939.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/>)

Author: Kirthi Devleker

Published: 2026-07-21T18:30:00Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [AI Platform](<https://devfeed.tech/topics/ai-platform.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [networking](<https://devfeed.tech/topics/networking.md>), [deepseek](<https://devfeed.tech/topics/deepseek.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [featured](<https://devfeed.tech/tags/featured.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [gb300-nvl72](<https://devfeed.tech/tags/gb300-nvl72.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-techniques](<https://devfeed.tech/tags/llm-techniques.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [megatron](<https://devfeed.tech/tags/megatron.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [performance](<https://devfeed.tech/tags/performance.md>), [top-stories](<https://devfeed.tech/tags/top-stories.md>), [train](<https://devfeed.tech/tags/train.md>), [training-ai-models](<https://devfeed.tech/tags/training-ai-models.md>)

### AI overview

The article explains how NVIDIA GB300 NVL72 achieved a world record for DeepSeek-V3 671B mixture-of-experts pre-training. It focuses on the communication demands of MoE models, including all-to-all traffic between GPUs, and the need for tightly coupled scale-up and predictable scale-out networking to sustain delivered training performance.

### Source excerpt

Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token...

## Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading

DevFeed: [Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading](<https://devfeed.tech/articles/reducing-high-bandwidth-memory-bottlenecks-in-jax-based-llm-training-with-host-offloading-6925.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/reducing-high-bandwidth-memory-bottlenecks-in-jax-based-llm-training-with-host-offloading/>)

Author: Tanya Lenz

Published: 2026-07-10T18:17:40Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gb200](<https://devfeed.tech/tags/gb200.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grace-cpu](<https://devfeed.tech/tags/grace-cpu.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-techniques](<https://devfeed.tech/tags/llm-techniques.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvl72](<https://devfeed.tech/tags/nvl72.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>), [simulation-modeling-design](<https://devfeed.tech/tags/simulation-modeling-design.md>), [training-ai-models](<https://devfeed.tech/tags/training-ai-models.md>), [vera-rubin](<https://devfeed.tech/tags/vera-rubin.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

This article explains how host offloading in JAX-based large language model training reduces GPU high-bandwidth memory pressure by moving selected activations to pinned host memory and streaming them back during the backward pass. It discusses activation-transfer overlap, NVIDIA Grace Blackwell and GB200 NVL72 systems, and experiments involving Llama 3.1 405B and DeepSeek-V3 671B.

### Source excerpt

Large language model (LLM) training workloads increasingly run into GPU memory limits before compute is fully used. Model weights, gradients, optimizer states,...

## Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

DevFeed: [Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism](<https://devfeed.tech/articles/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism-6815.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism/>)

Author: Michelle Horton

Published: 2026-07-06T21:44:23Z

Content type: article

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [Network](<https://devfeed.tech/topics/network.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [availability](<https://devfeed.tech/tags/availability.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [data-center-cloud](<https://devfeed.tech/tags/data-center-cloud.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [featured](<https://devfeed.tech/tags/featured.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-techniques](<https://devfeed.tech/tags/llm-techniques.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [network](<https://devfeed.tech/tags/network.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scale](<https://devfeed.tech/tags/scale.md>), [training](<https://devfeed.tech/tags/training.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

This article explains how Nonuniform Tensor Parallelism can improve Goodput in large-scale LLM training by adapting tensor parallelism to changing GPU availability and overlapping data resharding. The experimental approach aims to reduce interruptions, lost throughput, and computational waste in tightly interconnected GPU clusters.

### Source excerpt

Training LLMs at massive scale brings unique infrastructure challenges, especially as jobs span thousands of GPUs and run for extended periods. The longer these...

## Entropix dynamically adapts LLM execution strategies based on model signals

DevFeed: [Entropix dynamically adapts LLM execution strategies based on model signals](<https://devfeed.tech/articles/why-is-entropix-important-33441.md>)

Original publisher: [Read original article](<https://timkellogg.me/blog/2024/10/20/entropix-part2>)

Published: 2024-10-20T00:00:00Z

Content type: opinion

Language: en

Sources: [Tim Kellogg](<https://devfeed.tech/sources/tim-kellogg.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Automated prompt engineering](<https://devfeed.tech/topics/automated-prompt-engineering.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [JIT](<https://devfeed.tech/topics/jit.md>)

Tags: [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-techniques](<https://devfeed.tech/tags/llm-techniques.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

The article explains entropix as a proposed framework for LLM execution that uses signals from a model's internal state to switch dynamically among techniques such as chain-of-thought prompting and parameter adjustments. It presents this as an approach to automate some prompt-engineering decisions and notes that the current implementation relies on crude heuristics, with reinforcement learning being explored as a replacement.

### Source excerpt

The buzz is there! The cooks are cooking! But what is entropix? I wrote a fluff piece here explaining it, but the most comprehensive and complete description of entropix is here.