# Inference Performance Published articles for Inference Performance. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin DevFeed: [How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-15T16:55:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Groq 3 LPX](), [LPX](), [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [groq](), [Inference](), [Low Latency](), [long-context]() Tags: [agentic-ai-generative-ai](), [ai-factory](), [ai-inference](), [ai-platform](), [drive](), [dsx](), [groq](), [groq-3-lpx](), [inference](), [inference-performance](), [latency](), [low-latency](), [lpx](), [nvidia-vera-rubin](), [nvl72](), [performance](), [power-management](), [vera-rubin](), [vera-rubin-nvl72]() ### AI overview This NVIDIA developer article explains how Groq 3 LPX uses deterministic execution across 256 LPU chips to support low-latency inference on NVIDIA Vera Rubin. It describes compiler-scheduled execution and power-management techniques including Preemptive Power and Clock Period Synthesis. ### Source excerpt Power is a defining constraint for AI factories. As AI workloads demand a full compute platform to serve them, each component of that platform must maximize... ## How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra DevFeed: [How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-10T16:55:32Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Inference Performance](), [Language models]() Tags: [agentic-ai-generative-ai](), [benchmark](), [build-ai-agents](), [cache](), [developer-tools-techniques](), [gpu](), [inference](), [inference-performance](), [latency](), [mamba](), [mixture-of-experts](), [nemotron](), [nim](), [nvidia-blackwell](), [optimization]() ### AI overview The article explains how NVIDIA NIM bundles serving optimizations to improve throughput for Nemotron 3 Ultra while meeting latency targets on GPU infrastructure. ### Source excerpt Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as... ## Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson DevFeed: [Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-04T16:21:04Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Jetson](), [model-deployment](), [Inference](), [Inference Performance](), [Optimization](), [Mixture of Experts (MoE)]() Tags: [agentic-ai-generative-ai](), [edge](), [edge-computing](), [how-to](), [inference](), [inference-performance](), [jetpack](), [jetson](), [jetson-orin](), [mixture-of-experts-moe](), [nvfp4](), [optimization](), [physical-ai](), [reasoning](), [thor](), [tutorial](), [vllm]() ### AI overview A tutorial on deploying and optimizing compact reasoning and agentic AI models on NVIDIA Jetson. It covers choosing models, improving inference with NVFP4 quantization and speculative decoding, serving example models with vLLM, and validating a configuration for a workload. ### Source excerpt Running reasoning and agentic AI at the edge has been harder than it needs to be. Until recently, models capable of multi-step reasoning were too large to run... ## Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference DevFeed: [Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-02T16:04:19Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Inference](), [Large Language Model]() Tags: [accelerate](), [agentic-ai-generative-ai](), [ai](), [ai-inference](), [ai-models](), [batch](), [data-center-cloud](), [developer-tools-techniques](), [how-to](), [inference](), [inference-performance](), [llm](), [llms](), [nvfp4](), [training-ai-models]() ### AI overview The article explains speculative decoding as a way to speed up LLM inference while preserving standard-decoding outputs. A smaller draft model proposes several tokens, which the larger target model verifies in parallel; it also defines draft and acceptance lengths and gives a speedup formula. ### Source excerpt This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and... ## How to Size GPUs for AI Inference and TCO Without Overspending DevFeed: [How to Size GPUs for AI Inference and TCO Without Overspending]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-01T15:00:00Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [AI Inference](), [GPU](), [Inference Performance](), [Large Language Model](), [cloud-infrastructure](), [Optimization]() Tags: [agentic-ai-generative-ai](), [ai-factory](), [ai-inference](), [concurrency](), [cost](), [data-center-cloud](), [gpu](), [inference-performance](), [latency](), [llm](), [mlops](), [quantization]() ### AI overview A practical guide to sizing GPU infrastructure for AI inference workloads while balancing latency, concurrency, model choice, deployment strategy, and total cost of ownership. ### Source excerpt The surge in AI adoption is transforming everything from chatbots to content generation. Still, a common pain point remains: How can organizations confidently... ## How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin DevFeed: [How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-08-24T15:00:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [d-matrix](), [Vera Rubin](), [Inference Performance](), [long-context](), [Benchmark](), [Artificial Intelligence](), [gemma4](), [systems](), [Cache]() Tags: [agent](), [agentic](), [agentic-ai-generative-ai](), [ai](), [ai-factory](), [ai-inference](), [benchmark](), [cache](), [collective](), [data-center-cloud](), [developer-tools-techniques](), [groq](), [groq-3-lpx](), [inference-performance](), [long-context](), [low-latency-inference](), [lpx](), [nvidia](), [nvidia-vera](), [performance](), [rubin-gpu](), [speed](), [systems](), [tokens](), [training-ai-models](), [vera-rubin](), [vera-rubin-nvl72]() ### AI overview NVIDIA Groq 3 LPX, paired with Vera Rubin NVL72, delivers high-interactivity AI inference for long-context workloads. A reported benchmark measured 3,431 output tokens per second on Gemma 4 31B with a 100K context. ### Source excerpt NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the... ## 🍔🧠 What's inside an LLM's KV cache DevFeed: [🍔🧠 What's inside an LLM's KV cache]() Original publisher: [Read original article]() Author: Alexandre Zajac Published: 2026-08-10T15:30:59Z Content type: article Language: en Sources: [Hungry Minds]() Topics: [Cache](), [Caching](), [Large Language Model](), [Inference Performance](), [long-context](), [Inference]() Tags: [cache](), [caching](), [inference](), [inference-performance](), [latency](), [llm](), [models](), [performance](), [quantization]() ### AI overview The article explains that the KV cache stores key and value vectors for input tokens during LLM inference. Its memory use and bandwidth demands grow with context length and batch size, creating a bottleneck for long-context serving. It discusses grouped-query attention, latent attention, KV cache quantization, paged attention, and prefix caching as ways to reduce memory use, fragmentation, cost, or latency. ### Source excerpt PLUS: Google's AI depth decline 🧠, Real-time agent load balancing ⚡, Semantic search ranking at scale 👨💻 ## Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference DevFeed: [Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-07-31T22:16:17Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Inference Performance](), [long-context](), [Artificial Intelligence](), [GPU](), [GPU optimization](), [Large Language Model](), [Nvidia]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-inference](), [data-center-cloud](), [developer-tools-techniques](), [featured](), [gpu](), [inference](), [inference-performance](), [long-context](), [memory](), [model-architecture](), [nvidia](), [performance]() ### AI overview This article examines how co-designing dense attention with GPU execution can improve throughput and interactivity for long-context inference. It analyzes group size, head dimension, sequence length, and the different compute and memory behavior of prefill and decode, including the effects of speculative decoding and prefix caching. ### Source excerpt As agentic and long-context workloads become common, the context lengths increase and attention consumes a larger share of inference time (Figure 1). Because... ## ModelExpress: Distributing Model Artifacts at the Speed of Light DevFeed: [ModelExpress: Distributing Model Artifacts at the Speed of Light]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-24T16:45:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [model-deployment](), [GPU](), [P2P](), [Amazon S3](), [autoscaling](), [Caching](), [cloud-infrastructure]() Tags: [agentic-ai-generative-ai](), [ai-foundation-models](), [ai-inference](), [autoscaling](), [data-center-cloud](), [developer-tools-techniques](), [dynamo-triton](), [featured](), [gpu](), [inference](), [inference-performance](), [jit](), [llms](), [local](), [model](), [s3](), [scale](), [speed](), [streaming]() ### AI overview NVIDIA ModelExpress (MX) reduces model-serving startup time by locating compatible model weights at the fastest available source. It prioritizes direct GPU-to-GPU transfer over P2P RDMA from an existing serving replica, while using object or local storage for initial bootstrapping and fallback. The article also describes reusing JIT kernel caches and distributing reinforcement-learning weight updates. ### Source excerpt Every byte moved has a cost. As model checkpoints grow to hundreds of gigabytes or even a terabyte, that cost adds up quickly. To make things even worse, moving... ## Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI DevFeed: [Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI]() Original publisher: [Read original article]() Author: Eduardo Alvarez Published: 2026-07-21T18:15:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [GPU](), [Nvidia](), [Artificial Intelligence](), [Inference](), [Inference Performance](), [long-context](), [datacenter](), [Transformer](), [Deployment]() Tags: [accelerate](), [agentic](), [agentic-ai](), [agentic-ai-generative-ai](), [ai](), [ai-agent](), [ai-factory](), [ai-inference](), [architecture](), [blackwell](), [cache](), [compute](), [data-center](), [data-center-cloud](), [developer-tools-techniques](), [dsx](), [efficiency](), [featured](), [gpu](), [inference](), [inference-performance](), [latency](), [long-context](), [memory](), [mixture-of-experts-moe](), [nvfp4](), [nvidia](), [rubin-gpu](), [scale](), [tensor-cores](), [top-stories](), [training](), [vera-rubin]() ### AI overview This article examines the NVIDIA Rubin GPU architecture and its co-designed Vera Rubin platform for agentic AI inference. It describes how Tensor Cores, HBM4 memory, the Transformer Engine, NVFP4 performance, cache, decoding, and scale-up systems address throughput, latency, long-context execution, and rack-scale deployment. ### Source excerpt What began as discrete AI model training and human-facing chat interfaces has evolved into always-on AI factories dedicated to producing intelligence at scale.... ## AI Model Co-Design: Hardware-Friendly LLM Design DevFeed: [AI Model Co-Design: Hardware-Friendly LLM Design]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-10T16:36:02Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [AI Chat]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-inference](), [data-center-cloud](), [deployment](), [design](), [developer-tools-techniques](), [developers](), [featured](), [hardware](), [inference-performance](), [latency](), [llm](), [llms](), [model](), [performance](), [training](), [training-ai-models](), [vlms]() ### AI overview A practical primer on designing LLMs for modern hardware while balancing accuracy, throughput, and responsiveness. It explains how context length and latency or throughput goals change the importance of attention, feed-forward layers, and parallelism. ### Source excerpt AI performance comes down to three dimensions: Accuracy: How well the model reasons and produces outputs Throughput: How many tokens per second a... ## Native-speed vLLM transformers modeling backend DevFeed: [Native-speed vLLM transformers modeling backend]() Original publisher: [Read original article]() Author: Harry Mellor; Lysandre Published: 2026-07-08T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vllm](), [Inference Performance](), [Inference](), [LLMs](), [Machine learning](), [Open Source]() Tags: [architecture](), [batching](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [mixture-of-experts](), [mlx](), [open-source](), [optimization](), [sglang](), [transformers](), [vllm]() ### AI overview This article describes the transformers modeling backend for vLLM and its role in serving Hugging Face models without requiring model authors to port their implementations. It explains how transformers supplies modeling code while vLLM contributes optimized inference techniques, and outlines comparisons with vLLM's native implementations across several Qwen3 model configurations. It also discusses inference-performance factors including GPU parallelization, compilation, fused kernels, and custom attention kernels. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Hardware-Rooted AI Security That Won't Slow You Down DevFeed: [Hardware-Rooted AI Security That Won't Slow You Down]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-02T21:25:42Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Cryptography](), [Inference Performance]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-agent](), [ai-inference](), [ai-security](), [benchmarks](), [blackwell](), [cloud-services](), [code-software-generation](), [data-center-cloud](), [dynamo-triton](), [encryption](), [featured](), [gpu](), [hardware](), [inference-performance](), [latency](), [nvlink](), [security](), [security-for-ai](), [software-defined-data-center](), [tensorrt](), [tensorrt-llm](), [trustworthy-ai-cybersecurity]() ### AI overview An overview of NVIDIA Confidential Computing for protecting AI inference workloads, including hardware-rooted attestation and encryption. It reports benchmark results of up to 98% of the inference performance of configurations without confidential-computing security. ### Source excerpt AI has transformed how organizations operate, driving unprecedented levels of productivity and innovation. However, AI adoption can be impeded by concerns... ## NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency DevFeed: [NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency]() Original publisher: [Read original article]() Author: Waverly Swinton Published: 2026-03-19T22:13:37Z Content type: release Language: en Sources: [DigitalOcean]() Topics: [Dynamo](), [Digital Ocean](), [Inference Performance](), [Artificial Intelligence](), [Generative AI](), [Inference](), [Cloud](), [GB200](), [sglang](), [TensorRT-LLM](), [vllm](), [Kubernetes]() Tags: [ai-ml](), [cloud](), [cost](), [deployment](), [digitalocean](), [dynamo](), [gb200](), [inference](), [inference-performance](), [kubernetes](), [latency](), [llm](), [load-balancing](), [nvidia](), [product-updates](), [routing](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview NVIDIA Dynamo 1.0 is now available to DigitalOcean customers as an inference service framework for large-scale generative AI workloads. The article describes claimed performance improvements on NVIDIA GB200 NVL systems, cost-efficiency benefits, deployment options, and features including GPU-aware routing and disaggregated serving. ### Source excerpt NVIDIA Dynamo 1.0, which was released on Monday at NVIDIA GTC, is now available to DigitalOcean customers to help drive performance enhancements and cost efficiency. NVIDIA Dynamo 1.0 offers a 7x inference performance increase on NVIDIA GB200 NVL systems, and by pairing it with DigitalOcean's Agentic Inference Cloud, customers can achieve higher performance at lower costs while benefiting from seamless deployment. Working together, DigitalOcean's optimizations with NVIDIA have already achieved a 67% cost savings for customers like Workato, and this new generation of Dynamo can unlock even greater gains for businesses who run production-grade agentic workflows. DigitalOcean customers can get access to NVIDIA Dynamo 1.0 as a container image that can be run on a Droplet or can deploy directly on DigitalOcean Kubernetes with an inference runtime (vLLM, SGlang, TensorRT). What is NVIDIA Dynamo 1.0? NVIDIA Dynamo is a cutting-edge, high-performance inference service framework specifically designed to accelerate and optimize large-scale generative AI and inference models. Dynamo is an orchestration layer that sits above engines like vLLM, SGLang, and NVIDIA TensorRT-LLM. Think of it as the distributed traffic controller for your GPU fleet, seamlessly orchestrating GPU and memory resources across a cluster and reducing bottleneck by intelligently routing requests Key technical breakthroughs offered by Dynamo 1.0 include: 7x Performance Boost: When paired with NVIDIA Blackwell Ultra GPUs, Dynamo can increase inference performance by up to 7x, significantly lowering your cost per token. KV-Aware Routing: Instead of simple round-robin load balancing, Dynamo routes requests to the specific GPUs that already have the relevant "memory" from previous turns of a conversation. Disaggregated Serving: Dynamo splits the "prefill" (reading the prompt) and "decode" (generating the answer) phases across different GPUs to maximize utilization and reduce latency. Memory Offloading: The KV B ## GTX 1080 Ti for Local LLM DevFeed: [GTX 1080 Ti for Local LLM]() Original publisher: [Read original article]() Published: 2026-02-28T23:33:54Z Content type: article Language: en Sources: [Ariya Hidayat]() Topics: [GPU](), [LLMs](), [llama.cpp](), [Nvidia](), [CUDA](), [Inference Performance](), [Retrieval Augmented Generation (RAG)]() Tags: [cmake](), [command-line](), [cuda](), [gcc](), [gemma](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [nvidia]() ### AI overview This article evaluates the eight-year-old NVIDIA GTX 1080 Ti as an affordable option for running local LLMs. It explains how its 11 GB of VRAM can support quantized models and relevant context windows, then describes measuring prompt-processing and token-generation performance with CUDA-enabled llama.cpp. ### Source excerpt Despite being over eight years old, the NVIDIA GTX 1080 Ti remains a compelling choice for enthusiasts keen on running LLM locally. ## Technical Deep Dive: How DigitalOcean and AMD Delivered a 2x Production Inference Performance Increase for Character.ai DevFeed: [Technical Deep Dive: How DigitalOcean and AMD Delivered a 2x Production Inference Performance Increase for Character.ai]() Original publisher: [Read original article]() Author: Karnik Modi Published: 2026-01-13T12:30:00Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Artificial Intelligence](), [Digital Ocean](), [GPU](), [Inference](), [Inference Performance](), [GPU optimization](), [moe](), [Kubernetes](), [model-deployment](), [PyTorch](), [qwen]() Tags: [ai](), [digitalocean](), [engineering](), [gpu](), [inference](), [inference-performance](), [infrastructure](), [kubernetes](), [mixture-of-experts](), [news](), [performance](), [platforms](), [qwen](), [technical]() ### AI overview This technical deep dive describes how Character.ai, AMD, and DigitalOcean optimized inference for the Qwen3-235B Instruct FP8 model on AMD Instinct MI300X and MI325X GPUs. The reported optimizations, including parallelization, FP8 execution paths, optimized kernels, topology-aware allocation, and Kubernetes orchestration, produced up to a 2x improvement in production request throughput under stated latency and concurrency constraints. ### Source excerpt Background: How Character.ai worked with DigitalOcean and AMD to optimize performance Character.ai, a leading AI entertainment platform with about 20 million worldwide users, wanted to optimize GPU performance and achieve lower inference costs for its application, which requires low-latency performance at large scale. They approached DigitalOcean and AMD in order to achieve this goal. Working closely together, the Character.ai, AMD, and DigitalOcean teams optimized AMD Instinct™ MI300X and MI325X GPU platforms, resulting in a 2x production inference throughput. In optimized configurations, DigitalOcean delivered high request density per node while maintaining exceptional p90 responsiveness for initial token and sustained token generation throughput, outperforming prior deployments on generic, non-optimized GPU infrastructure. These gains were achieved through platform-level optimizations, including clever parallelization strategies for large Mixture-of-Experts models, efficient FP8 execution paths, optimized kernels with AITER, topology-aware GPU allocation, and production-ready Kubernetes orchestration through DigitalOcean Kubernetes (DOKS). Together, these capabilities allowed Character.ai to scale inference predictably without increasing operational burden. In this post, we will explore the specific orchestration and tuning strategies that made these gains possible. Technical deep dive overview Character.ai leverages multiple models like Qwen, Mistral and more to power their applications. This document is focused on how we optimized the Qwen3-235B Instruct FP8 model on a cluster of DigitalOcean featuring AMD Instinct GPUs. This workload was migrated from a generic, non-optimized setup on other providers to AMD Instinct™ MI325X platform on DigitalOcean, and following the outlined optimizations we were able to achieve up to a 2x improvement in request throughput (QPS) under strict latency and concurrency constraints. The Character.ai team has a demanding workload, ## Generate Apps Locally for Free: App.build Now Supports Open Source Models DevFeed: [Generate Apps Locally for Free: App.build Now Supports Open Source Models]() Original publisher: [Read original article]() Author: Arseni Kravchenko Published: 2025-08-11T19:41:53Z Content type: release Language: en Sources: [Blog -- Neon Docs]() Topics: [AI-assisted coding](), [Inference Performance]() Tags: [agent](), [api](), [apps](), [cloud](), [coding](), [hardware](), [inference](), [inference-performance](), [llms](), [local](), [models](), [open-source](), [openai](), [product]() ### AI overview App.build adds support for running open-weight LLMs through Ollama, LMStudio, and OpenRouter to generate applications without depending on cloud APIs. The article discusses cost, rate-limit, privacy, and local-hardware considerations, including example inference speeds. ### Source excerpt App.build now supports open weights LLMs via Ollama, LMStudio, and OpenRouter - enabling you to generate complete applications end-to-end without cloud API dependency or associated costs. Why Run App.build Locally? Zero API costs Cloud LLM APIs can become expensive fast during ex... ## Transformers backend integration in SGLang DevFeed: [Transformers backend integration in SGLang]() Original publisher: [Read original article]() Author: Yineng Zhang; Ke Bao; Lianmin; Jin Pan; Marc Sun Published: 2025-06-23T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [Inference Performance]() Tags: [api](), [backend](), [deployment](), [hugging-face](), [inference](), [inference-performance](), [integration](), [latency](), [model](), [sglang](), [transformers]() ### AI overview SGLang adds a Transformers backend that supports high-performance inference for Transformers-compatible models, including automatic fallback for models without native SGLang support. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Accelerating Protein Language Model ProtST on Intel Gaudi 2 DevFeed: [Accelerating Protein Language Model ProtST on Intel Gaudi 2]() Original publisher: [Read original article]() Author: Julien Simon; Jiqing.Feng; Santiago Miret; Xinyu Yuan; Yi Wang; Matrix Yao; Minghao Xu; Ke Ding Published: 2024-07-03T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [Large Language Model](), [Inference](), [Inference Performance](), [Fine-tuning](), [intel](), [Open Source](), [AI, ML & Data Engineering]() Tags: [accelerators](), [artificial-intelligence](), [batch](), [datasets](), [fine-tuning](), [hugging-face](), [inference](), [inference-performance](), [intel](), [language-models](), [llm](), [model](), [nvidia](), [optimum](), [partnerships](), [pcie](), [precision](), [transformers](), [zero-shot]() ### AI overview This tutorial explains how to run inference and fine-tune ProtST, a multimodal protein language model, using Intel Gaudi 2 accelerators and the Optimum for Intel Gaudi open-source library. It compares ProtST inference on NVIDIA A100 and Gaudi 2, reporting identical accuracy and 1.76x faster inference on Gaudi 2. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Faster Dynamically Quantized Inference with XNNPack DevFeed: [Faster Dynamically Quantized Inference with XNNPack]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2024-04-09T16:00:00Z Content type: article Language: en Sources: [The TensorFlow Blog]() Topics: [Inference Performance](), [quantization](), [TensorFlow Lite](), [cpu](), [Machine learning](), [model-deployment]() Tags: [ai](), [announcement](), [article](), [cpu](), [inference](), [inference-performance](), [learn](), [ml](), [performance](), [precision](), [quantization](), [range](), [scale](), [tensorflow](), [tensorflow-lite](), [tensors]() ### AI overview This article explains how dynamic range quantization for XNNPack's Fully Connected and Convolution 2D operators improves CPU inference performance in TensorFlow Lite. The approach quantizes weights to 8-bit integers during model conversion and dynamically quantizes activations during inference, delivering most of the performance benefits of full quantization while retaining higher overall accuracy. The article reports a fourfold performance improvement over the single-precision baseline and describes how this can enable AI-powered features on older and lower-tier devices. ### Source excerpt Posted by Alan Kelly, Software Engineer We are excited to announce that XNNPack's Fully Connected and Convolution 2D operators now support dynamic range quantization. XNNPack is TensorFlow Lite's CPU backend and CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority. We quadrupled inference performance in TensorFlow Lite's XNNPack backend compared to the single precision baseline by adding support for dynamic range quantization to the Fully Connected and Convolution operators. This means that more AI powered features may be deployed to older and lower tier devices. Previously, XNNPack offered users the choice between either full integer quantization, where the weights and activations are stored as signed 8-bit integers, or half-precision (fp16) or single-precision (fp32) floating-point inference. In this article we demonstrate the benefits of dynamic range quantization. Dynamic Range Quantization Dynamically quantized models are similar to fully-quantized models in that the weights for the Fully Connected and Convolution operators are quantized to 8-bit integers during model conversion. All other tensors are not quantized, they remain as float32 tensors. During model inference, the floating-point layer activations are converted to 8-bit integers before being passed to the Fully Connected and Convolution operators. The quantization parameters (the zero point and scale) for each row of the activation tensor are calculated dynamically based on the observed range of activations. This maximizes the accuracy of the quantization process as the activations make full use of the 8 quantized bits. In fully-quantized models, these parameters are fixed during model conversion, based on the range of the activation values observed using a representative dataset. The second difference between full quantization and dynamic range quantization is that the output of the Fully ## Half-precision Inference Doubles On-Device Inference Performance DevFeed: [Half-precision Inference Doubles On-Device Inference Performance]() Original publisher: [Read original article]() Author: TensorFlow Blog (noreply@blogger.com) Published: 2023-11-29T18:00:00Z Content type: release Language: en Sources: [The TensorFlow Blog]() Topics: [Inference Performance](), [On-device AI](), [LiteRT](), [cpu](), [AI, ML & Data Engineering]() Tags: [ai](), [announcement](), [benchmarks](), [cpu](), [how-to](), [inference](), [inference-performance](), [learn](), [on-device](), [tensorflow-lite]() ### AI overview TensorFlow Lite and XNNPack add generally available FP16 inference for supported ARM CPUs, reporting close to 2x speedups for floating-point models in production and benchmarked mobile and laptop environments. ### Source excerpt Posted by Marat Dukhan and Frank Barchard, Software Engineers CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority, and we are excited to announce that we doubled floating-point inference performance in TensorFlow Lite's XNNPack backend by enabling half-precision inference on ARM CPUs. This means that more AI powered features may be deployed to older and lower tier devices. Traditionally, TensorFlow Lite supported two kinds of numerical computations in machine learning models: a) floating-point using IEEE 754 single-precision (32-bit) format and b) quantized using low-precision integers. While single-precision floating-point numbers provide maximum flexibility and ease of use, they come at the cost of 4X overhead in storage and memory and exhibit a performance overhead compared to 8-bit integer computations. In contrast, half-precision (FP16) floating-point numbers pose an interesting alternative balancing ease-of-use and performance: the processor needs to transfer twice fewer bytes and each vector operation produces twice more elements. By virtue of this property, FP16 inference paves the way for 2X speedup for floating-point models compared to the traditional FP32 way. For a long time FP16 inference on CPUs primarily remained a research topic, as the lack of hardware support for FP16 computations limited production use-cases. However, around 2017 new mobile chipsets started to include support for native FP16 computations, and by now most mobile phones, both on the high-end and the low-end. Building upon this broad availability, we are pleased to announce the general availability for half-precision inference in TensorFlow Lite and XNNPack. Performance Improvements Half-precision inference has already been battle-tested in production across Google Assistant, Google Meet, YouTube, and ML Kit, and demonstrated close to 2X speedups across a wide range of ne