# Post-training optimization

Machine-learning techniques applied to trained models, such as quantization, pruning, and distillation, to improve deployment efficiency or model behavior.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy

DevFeed: [Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy](<https://devfeed.tech/articles/understanding-w8a8-int8-llm-quantization-half-the-size-better-performance-same-accuracy-12347.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/09/07/understanding-w8a8-int8-llm-quantization>)

Author: Sana Fayyaz

Published: 2026-09-07T07:01:34Z

Content type: article

Language: en

Sources: [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [llama](<https://devfeed.tech/topics/llama.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [compression](<https://devfeed.tech/tags/compression.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

This article explains INT8 W8A8 quantization for Llama 3.1 8B using SmoothQuant and GPTQ. It describes how quantization reduces memory requirements and can improve serving performance while preserving model accuracy, including a reported 46% reduction in model size and planned benchmark testing with vLLM and llm-compressor.

### Source excerpt

Large language models are expensive to serve. A model like Llama 3.1 8B in Bfloat16 (BF16) precision occupies roughly 15 GB of GPU memory. In BF16, each of the 8 billion parameters takes 2 bytes to store, which adds up to roughly 15 GB for the weights--and that's not all. The GPU needs memory for the key-value (KV) cache to store context for active requests, alongside intermediate tensor outputs (activations, as we call them) generated during inference. The post Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy appeared first on Red Hat Developer.

## Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost

DevFeed: [Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost](<https://devfeed.tech/articles/gisting-compressing-llm-agent-context-to-throughput-and-cost-1403.md>)

Original publisher: [Read original article](<https://shopify.engineering/gisting>)

Author: Cody Mazza-Anthony

Published: 2026-08-19T14:32:58Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [Low-Latency Inference](<https://devfeed.tech/topics/low-latency-inference.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [GraphQL](<https://devfeed.tech/topics/graphql.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

Gisting compresses an LLM agent's system prompt into learned gist tokens, preserving prediction quality while reducing inference latency, increasing throughput, and lowering GPU requirements.

### Source excerpt

Gisting compresses context into a set of learned tokens, preserving its quality while making the model faster and cheaper.

## Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer

DevFeed: [Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer](<https://devfeed.tech/articles/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer-6811.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer/>)

Author: Tanya Lenz

Published: 2026-08-17T18:12:48Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Mamba](<https://devfeed.tech/topics/mamba.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [compute](<https://devfeed.tech/tags/compute.md>), [developer-tools-techniques](<https://devfeed.tech/tags/developer-tools-techniques.md>), [developers](<https://devfeed.tech/tags/developers.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [featured](<https://devfeed.tech/tags/featured.md>), [latency](<https://devfeed.tech/tags/latency.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [megatron](<https://devfeed.tech/tags/megatron.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [model-optimizer](<https://devfeed.tech/tags/model-optimizer.md>), [models](<https://devfeed.tech/tags/models.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open](<https://devfeed.tech/tags/open.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [speed](<https://devfeed.tech/tags/speed.md>), [training](<https://devfeed.tech/tags/training.md>), [training-ai-models](<https://devfeed.tech/tags/training-ai-models.md>)

### AI overview

This tutorial explains how quantization-aware distillation (QAD) creates the Nemotron 3.5 Lightning NVFP4 checkpoint using NVIDIA Model Optimizer. It covers post-training quantization, teacher-student distillation, and evaluation, showing how QAD can recover accuracy while reducing memory usage and increasing throughput.

### Source excerpt

Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models, developers can find...

## Making Knowledge Distillation Cheap Enough to Run at Scale

DevFeed: [Making Knowledge Distillation Cheap Enough to Run at Scale](<https://devfeed.tech/articles/making-knowledge-distillation-cheap-enough-to-run-at-scale-7021.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation>)

Author: Antonio Tiene; Iker García-Ferrero; Ali Hashemi; Bakbergen Ryskulov

Published: 2026-08-10T10:05:36Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [gpt-oss](<https://devfeed.tech/topics/gpt-oss.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [caching](<https://devfeed.tech/tags/caching.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cost](<https://devfeed.tech/tags/cost.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-oss](<https://devfeed.tech/tags/gpt-oss.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [research](<https://devfeed.tech/tags/research.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

This article presents an efficient approach to knowledge distillation for large language models. It caches the teacher model's top-K logits offline and uses a fused, memory-efficient chunked KL-divergence loss, avoiding simultaneous teacher-student residency and full vocabulary-by-sequence-length matrices. The changes reduce VRAM use and training cost, enabling long-context distillation on a single GPU and making larger-scale experimentation more practical.

### Source excerpt

A Blog post by Multiverse Computing on Hugging Face

## Welcome to the Falcon 3 Family of Open Models!

DevFeed: [Welcome to the Falcon 3 Family of Open Models!](<https://devfeed.tech/articles/welcome-to-the-falcon-3-family-of-open-models-7190.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/falcon3>)

Author: Falcon LLM TII UAE

Published: 2024-12-17T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Open Source](<https://devfeed.tech/topics/open-source.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Mamba](<https://devfeed.tech/topics/mamba.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [community](<https://devfeed.tech/tags/community.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [models](<https://devfeed.tech/tags/models.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [releases](<https://devfeed.tech/tags/releases.md>), [research](<https://devfeed.tech/tags/research.md>), [science](<https://devfeed.tech/tags/science.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Falcon3 is a family of open models designed to improve science, mathematics, and code capabilities across small and medium language-model scales. The article describes transformer-based and pure state-space variants, training and scaling methods, compact models produced with pruning and knowledge distillation, multiple quantized formats, and benchmark results against comparable models.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Blazing Fast SetFit Inference with 🤗 Optimum Intel on Xeon

DevFeed: [Blazing Fast SetFit Inference with 🤗 Optimum Intel on Xeon](<https://devfeed.tech/articles/blazing-fast-setfit-inference-with-optimum-intel-on-xeon-7473.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/setfit-optimum-intel>)

Author: Daniel Korat; Tom Aarsen; Oren Pereg; Moshe Wasserblat; Ella Charlaix; Abirami Prabhakaran

Published: 2024-04-03T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [community](<https://devfeed.tech/tags/community.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-collab](<https://devfeed.tech/tags/open-source-collab.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

A tutorial on accelerating SetFit inference on Intel Xeon CPUs with Optimum Intel and post-training quantization.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.