# Inference Performance The performance of systems serving trained machine-learning models, typically evaluated through latency, throughput, resource use, and cost. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor DevFeed: [TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-16T20:37:07Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Jetson AGX Thor Developer Kit](), [Inference Performance](), [benchmarking](), [TensorRT](), [Large Language Model](), [quantization](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [evaluation](), [jetson](), [jetson-agx-thor-developer-kit](), [llm](), [llm-benchmarking](), [mlperf](), [nvidia](), [performance](), [quantization](), [tensorrt](), [tensorrt-llm](), [thor]() ### AI overview This article reports that NVIDIA TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit for the MLPerf Inference v6.1 Edge Agentic benchmark. Using NVFP4 quantization, tree-based multi-token prediction, and KV cache reuse, it achieved 52.33 tokens per second and completed 1,007 turns in 24 minutes and 36 seconds, 6.4 times faster than the llama.cpp reference submission. ### Source excerpt AI agents are moving from cloud data centers to vehicles, robots, and other edge devices. Unlike a chatbot that answers a single prompt, an agent works through... ## NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut DevFeed: [NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut]() Original publisher: [Read original article]() Author: Zhihan Jiang Published: 2026-09-16T15:00:48Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [benchmarking](), [Inference Performance](), [Dynamo](), [vllm](), [Nvidia](), [TensorRT-LLM]() Tags: [ai-infrastructure](), [benchmarks](), [dynamo](), [efficiency](), [hardware](), [inference](), [mlperf](), [networking](), [nvidia](), [nvidia-vera-rubin](), [nvl72](), [optimization](), [performance](), [software](), [tensorrt](), [tensorrt-llm](), [vera-rubin-nvl72](), [vllm]() ### AI overview NVIDIA reports MLPerf Inference v6.1 preview results for Vera Rubin NVL72 and GB300 NVL72 systems. Vera Rubin NVL72 delivered up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x higher throughput on DeepSeek-R1, while a four-rack GB300 NVL72 submission achieved 99% scaling efficiency. The results used vLLM, NVIDIA Dynamo, and TensorRT-LLM. ### Source excerpt System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. [...] ## MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers DevFeed: [MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-16T15:00:00Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Benchmark](), [Inference Performance](), [Retrieval Augmented Generation (RAG)](), [agentic-coding](), [GPU](), [Vera Rubin NVL72](), [Nvidia](), [Vera Rubin]() Tags: [agentic-coding](), [ai](), [benchmark](), [enterprise](), [gpt-oss](), [gpu](), [inference](), [llama-cpp](), [numbers](), [nvidia](), [qwen3](), [rag]() ### AI overview MLCommons published MLPerf Inference v6.1 with record participation, two new inference tests, and peer-reviewed results for several newly covered accelerators. The release reports a 5.7x improvement in the best per-accelerator DeepSeek-R1 server result compared with v5.1. ### Source excerpt MLCommons has published MLPerf Inference v6.1, and the round sets a participation record with 30 submitting organizations and 486 datacenter and edge results. Two new tests join the suite: an End-to-End RAG pipeline for the datacenter and an Edge Agentic Inference benchmark for single-user devices, and the results carry the first peer-reviewed numbers for NVIDIA's The post MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers appeared first on StorageReview.com. ## Fujitsu MONAKA Server Brings 2nm 144-Core CPUs to Air-Cooled AI Inference, On Sale in November DevFeed: [Fujitsu MONAKA Server Brings 2nm 144-Core CPUs to Air-Cooled AI Inference, On Sale in November]() Original publisher: [Read original article]() Author: Lyle Smith Published: 2026-09-14T18:03:44Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [AI Inference](), [AI Infrastructure](), [Inference Performance](), [cpu](), [data centers](), [Confidential Computing](), [Arm](), [Hardware](), [NVLink](), [Nvidia]() Tags: [ai](), [ai-inference](), [ai-infrastructure](), [arm](), [benchmark](), [data-centers](), [enterprise](), [fujitsu](), [hardware](), [nvidia](), [nvlink]() ### AI overview Fujitsu is introducing MONAKA Servers built around its 2nm FUJITSU-MONAKA processor for AI inference in air-cooled data centers. The servers offer up to 144 CPU cores, matrix instructions, SVE2 vector processing, hardware-level confidential computing, and planned NVLink Fusion integration with NVIDIA GPUs. Fujitsu claims higher inference throughput and reduced cooling power consumption, but the article notes that supporting benchmark details are unavailable. ### Source excerpt Fujitsu is bringing its 2nm FUJITSU-MONAKA processor to AI infrastructure with a new server family designed to run AI inference in air-cooled data centers without requiring specialized liquid cooling. The MONAKA Server is designed, developed, and manufactured in Japan, with component and manufacturing traceability for sovereign AI deployments. The first MONAKA Servers will come in The post Fujitsu MONAKA Server Brings 2nm 144-Core CPUs to Air-Cooled AI Inference, On Sale in November appeared first on StorageReview.com. ## Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts DevFeed: [Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-14T16:23:21Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Inference](), [Inference Performance](), [Orchestration](), [GPU](), [Multi-tenancy](), [sglang](), [TensorRT](), [vllm](), [Security]() Tags: [ai](), [cache](), [concurrent](), [enterprise](), [gpu](), [inference](), [latency](), [nvme](), [orchestration](), [security]() ### AI overview Lightbits Labs is introducing Inferra, a KV cache orchestration engine for AI inference. It virtualizes GPU memory across DRAM and NVMe storage, preserving attention states for long-context and multi-session workloads. Lightbits claims up to 16 times more concurrent sessions, more than 100 times lower latency than recomputation, and context windows of up to 10 million tokens. Inferra supports vLLM, TensorRT, and SGLang and includes tiering, predictive prefetching, tenant isolation, and encrypted data transfer. ### Source excerpt Lightbits Labs, the company that invented NVMe over TCP, is moving into inference software with Inferra, a KV cache orchestration engine that makes its public debut tomorrow, September 15, at the AI Infra Summit in Santa Clara. The software virtualizes GPU memory across DRAM and NVMe storage tiers and turns the KV cache into a The post Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts appeared first on StorageReview.com. ## Chip Huyen explains how to cut inference costs without new hardware DevFeed: [Chip Huyen explains how to cut inference costs without new hardware]() Original publisher: [Read original article]() Author: Tim Koopmans Published: 2026-09-13T15:00:00Z Content type: article Language: en Sources: [The New Stack]() Topics: [Inference Performance](), [Low-Latency Inference](), [model-deployment](), [Optimization](), [Frontier Model](), [AI Engineering](), [Latency](), [math]() Tags: [ai](), [ai-agents](), [ai-engineering](), [ai-infrastructure](), [frontier-model](), [inference](), [low-latency](), [metrics](), [optimization](), [performance](), [post-contributed](), [reasoning](), [scylladb](), [sponsor-scylladb](), [sponsored](), [sponsored-post-contributed](), [tokens]() ### AI overview Chip Huyen explains why inference costs can outweigh one-time frontier-model training costs and outlines ways to optimize inference without new hardware. The article emphasizes latency metrics such as time to first token, time per output token, end-to-end latency, and goodput, especially for reasoning models. ### Source excerpt Last October, the P99 conference -- the online gathering for developers focused on high-performance, low-latency applications -- featured a cracking The post Chip Huyen explains how to cut inference costs without new hardware appeared first on The New Stack. ## How to use Google microbenchmarks for evaluating TPU performance DevFeed: [How to use Google microbenchmarks for evaluating TPU performance]() Original publisher: [Read original article]() Author: Junjie Qian; Chi Shuen Lee; Yu-Hsuan (Amy) Lin; Haixiong (Sean) Wang Published: 2026-09-12T11:04:33.891311Z Content type: tutorial Language: en Sources: [Google Developers Blog]() Topics: [Google AI](), [Inference Performance]() Tags: [architecture](), [benchmarks](), [compute](), [developers](), [google](), [guides](), [hardware](), [how-to](), [kernel](), [machine-learning](), [memory](), [mesh](), [metrics](), [model](), [open](), [open-source](), [performance](), [scale](), [software](), [tpu]() ### AI overview A tutorial on using Google's TPU microbenchmark suite to measure network, compute, memory, host-transfer, and attention performance. The results can establish a Roofline baseline and guide workload-specific optimization. ### Source excerpt Google's open-source TPU microbenchmark suite provides developers with granular performance metrics across Network, Compute, HBM, Host Transfer, and Attention components to validate real-world hardware capabilities. By leveraging these benchmarks to establish a Roofline model, engineers can accurately diagnose whether their machine learning workloads are compute-, memory-, or network-bound. This empirical baseline directly guides targeted software optimizations--such as kernel tuning, mesh sharding, and rematerialization--to maximize hardware utilization for large-scale model deployments. ## Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:58:09Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Large Language Model](), [Low-Latency Inference](), [Caching](), [Inference](), [Inference Performance]() Tags: [advanced-300](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [announcements](), [benchmarks](), [caching](), [inference](), [latency](), [llama](), [llm](), [routing](), [tensorrt-llm](), [vllm]() ### AI overview Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token. ### Source excerpt Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%. ## How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra DevFeed: [How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-10T16:55:32Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Inference Performance](), [Language models]() Tags: [agentic-ai-generative-ai](), [benchmark](), [build-ai-agents](), [cache](), [developer-tools-techniques](), [gpu](), [inference](), [inference-performance](), [latency](), [mamba](), [mixture-of-experts](), [nemotron](), [nim](), [nvidia-blackwell](), [optimization]() ### AI overview The article explains how NVIDIA NIM bundles serving optimizations to improve throughput for Nemotron 3 Ultra while meeting latency targets on GPU infrastructure. ### Source excerpt Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as... ## When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving DevFeed: [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-09T20:31:04Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Optimization](), [model-deployment](), [multimodal](), [Dynamo](), [Inference Performance](), [Embeddings](), [Mixture of Experts (MoE)](), [GPU](), [Open Source](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [computer-vision-video-analytics](), [developer-tools-techniques](), [dynamo](), [embeddings](), [gpu](), [inference](), [low-latency-inference](), [mixture-of-experts-moe](), [model-serving](), [multimodal](), [nvfp4](), [open-source](), [optimization]() ### AI overview This article explains when encode-prefill-decode disaggregation improves multimodal model serving. Using NVIDIA Dynamo, it separates vision encoding from LLM prefill and decode so the stages can batch, schedule, and scale independently. The approach is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, with reported gains of up to 5x in time to first token and 7x in end-to-end response time. ### Source excerpt Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... ## Qualcomm and Amazon Sign Multi-Generation Deal for Custom AI Inference Silicon and 1.6T Optical Interconnects DevFeed: [Qualcomm and Amazon Sign Multi-Generation Deal for Custom AI Inference Silicon and 1.6T Optical Interconnects]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-08T17:17:46Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Inference](), [Inference Performance](), [AI Infrastructure](), [Amazon Web Services](), [data centers](), [Chip design](), [networking](), [Amazon Bedrock](), [amazon]() Tags: [ai](), [ai-data-centers](), [ai-inference](), [ai-infrastructure](), [amazon-bedrock](), [aws](), [chip-design](), [cost](), [data-centers](), [dsp](), [efficiency](), [enterprise](), [networking](), [partnership](), [qualcomm]() ### AI overview Qualcomm Technologies and Amazon are collaborating across multiple generations to develop custom silicon for AWS AI data centers, primarily targeting AI inference. The agreement also covers 1.6T optical connectivity for data center networks and Qualcomm's use of AWS infrastructure, including Amazon Bedrock, for electronic design automation workloads. ### Source excerpt Qualcomm Technologies and Amazon have entered into a multi-generation collaboration to deliver customized silicon at scale for AWS's AI data centers, with AI inference as the primary target. The agreement pairs Qualcomm's power-efficient processing, silicon design, and system-level integration with Amazon's AI infrastructure, and is aimed at the compute, memory bandwidth, networking, and energy constraints The post Qualcomm and Amazon Sign Multi-Generation Deal for Custom AI Inference Silicon and 1.6T Optical Interconnects appeared first on StorageReview.com. ## Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson DevFeed: [Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-04T16:21:04Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Jetson](), [model-deployment](), [Inference](), [Inference Performance](), [Optimization](), [Mixture of Experts (MoE)]() Tags: [agentic-ai-generative-ai](), [edge](), [edge-computing](), [how-to](), [inference](), [inference-performance](), [jetpack](), [jetson](), [jetson-orin](), [mixture-of-experts-moe](), [nvfp4](), [optimization](), [physical-ai](), [reasoning](), [thor](), [tutorial](), [vllm]() ### AI overview A tutorial on deploying and optimizing compact reasoning and agentic AI models on NVIDIA Jetson. It covers choosing models, improving inference with NVFP4 quantization and speculative decoding, serving example models with vLLM, and validating a configuration for a workload. ### Source excerpt Running reasoning and agentic AI at the edge has been harder than it needs to be. Until recently, models capable of multi-step reasoning were too large to run... ## Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data DevFeed: [Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-03T16:22:15Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [AI Inference](), [Inference Performance](), [AI Infrastructure](), [model-serving](), [Open Source Models & Datasets](), [data centers](), [networking]() Tags: [ai](), [ai-inference](), [ai-infrastructure](), [architectures](), [data-centers](), [enterprise](), [hardware](), [inference](), [latency](), [networking](), [open-source](), [production]() ### AI overview Equinix Inference Exchange is a distributed AI inference platform that places NVIDIA compute and Together AI's open-model serving closer to enterprise data, users, and applications. It combines Equinix's interconnection infrastructure, NVIDIA hardware, and support for more than 200 open-source models to address latency, data sovereignty, networking complexity, and inference costs. ### Source excerpt Equinix has expanded its partnership with NVIDIA and entered a new collaboration with Together AI to launch Equinix Inference Exchange. Designed as a distributed AI inference architecture for enterprise deployments, the platform aims to shift compute workloads closer to core data repositories, end users, and operational applications. Announced alongside Equinix Fabric One at the Equinix The post Equinix Inference Exchange Brings NVIDIA Compute and 200+ Open Models Closer to Enterprise Data appeared first on StorageReview.com. ## Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026 DevFeed: [Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026]() Original publisher: [Read original article]() Author: Gerardo Delgado Published: 2026-09-03T16:00:59Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Local AI](), [Inference](), [Hardware](), [Inference Performance](), [Mixture of Experts (MoE)]() Tags: [agentic-ai](), [agents](), [ai](), [ai-inference](), [announcements](), [artificial-intelligence](), [dgx-spark](), [hardware](), [inference](), [llama-cpp](), [local-ai](), [mixture-of-experts-moe](), [nvidia](), [nvidia-rtx](), [open-source](), [rtx-ai-garage](), [rtx-spark](), [video](), [vllm]() ### AI overview NVIDIA announces local-AI updates at IFA 2026, including agent tooling, faster local inference, RTX Spark Windows PCs, and locally runnable models for agentic, coding, and video-generation workloads. ### Source excerpt Frontier intelligence is going local. At IFA 2026, NVIDIA, Microsoft and its partners are teaming up to provide faster inference and new tools that make agents easier to set up and run locally on NVIDIA hardware. New compact NVIDIA RTX Spark Windows PCs are also coming in October to give AI enthusiasts, developers and creators [...] ## Fast model loading for AI inference on Amazon EKS DevFeed: [Fast model loading for AI inference on Amazon EKS]() Original publisher: [Read original article]() Author: Sajjan Gundapuneedi Published: 2026-09-01T15:48:15Z Content type: article Language: en Sources: [Containers]() Topics: [Inference Performance](), [Amazon Elastic Kubernetes Service](), [Amazon S3](), [Language models]() Tags: [advanced-300](), [ai](), [ai-inference](), [amazon-eks](), [amazon-elastic-kubernetes-service](), [best-practices](), [cuda](), [gpu](), [inference](), [kubernetes](), [language-models](), [pytorch](), [s3](), [sglang](), [startup](), [vllm]() ### AI overview The article analyzes cold-start delays for AI inference pods on Amazon EKS. It finds that startup bottlenecks vary by model size: torch.compile dominates for smaller models, while loading weights from S3 to GPU memory dominates for larger models. Configuration changes to Run:ai Model Streamer reduce model-loading time on repeat launches. ### Source excerpt When you scale AI inference on Amazon EKS, every new pod must load model weights into GPU memory before serving traffic. We investigated where cold-start time goes and found two configuration-only changes to Run:ai Model Streamer that cut model startup time by 80-93% on subsequent launches, with no code changes. ## How to Size GPUs for AI Inference and TCO Without Overspending DevFeed: [How to Size GPUs for AI Inference and TCO Without Overspending]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-01T15:00:00Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [AI Inference](), [GPU](), [Inference Performance](), [Large Language Model](), [cloud-infrastructure](), [Optimization]() Tags: [agentic-ai-generative-ai](), [ai-factory](), [ai-inference](), [concurrency](), [cost](), [data-center-cloud](), [gpu](), [inference-performance](), [latency](), [llm](), [mlops](), [quantization]() ### AI overview A practical guide to sizing GPU infrastructure for AI inference workloads while balancing latency, concurrency, model choice, deployment strategy, and total cost of ownership. ### Source excerpt The surge in AI adoption is transforming everything from chatbots to content generation. Still, a common pain point remains: How can organizations confidently... ## IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining DevFeed: [IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining]() Original publisher: [Read original article]() Published: 2026-08-26T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [Language models](), [compression and generalization](), [Training AI Models](), [Inference Performance]() Tags: [compression](), [generative](), [inference](), [language-models](), [large-language-models](), [model](), [training]() ### AI overview This paper presents IDEA Prune, an integrated enlarge-and-prune pipeline for generative language model pretraining. It combines enlarged-model training, iterative structured pruning, and recovery under one cosine annealing learning-rate schedule, with experiments compressing 2.8B models to 1.3B. ### Source excerpt Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token efficiency compared to training target-size models from scratch. In this paper, we advocate incorporating enlarged model pretraining, which is often ignored in previous works, into pruning. We study the enlarge-and-prune pipeline as an integrated system to address two critical questions: whether it is worth pretraining an enlarged model even when the model is never deployed, and how to optimize the... ## Happy 20th Birthday, Amazon EC2 DevFeed: [Happy 20th Birthday, Amazon EC2]() Original publisher: [Read original article]() Author: Channy Yun (윤석찬) Published: 2026-08-25T16:11:02Z Content type: article Language: en Sources: [AWS News Blog]() Topics: [Amazon EC2](), [Inference Performance]() Tags: [ai-inference](), [amazon-cloudwatch](), [amazon-ec2](), [amazon-ec2-mac-instances](), [apple](), [aws](), [aws-inferentia](), [aws-local-zones](), [aws-outposts](), [aws-trainium](), [cloud](), [cloud-computing](), [general](), [generative-ai](), [graviton](), [infrastructure](), [intel](), [linux](), [news](), [nitro-system](), [performance]() ### AI overview AWS marks Amazon EC2's 20th anniversary by reviewing its evolution from a single Linux virtual-server instance to a global service with more than 1,200 instance types. The article highlights foundational services, expanded deployment options, and AWS-designed silicon for AI inference and training. ### Source excerpt On the 20th Anniversary, we recognize how AWS has continued to push the boundaries of what cloud computing can deliver, building custom silicon for general-purpose and AI workloads and expanding EC2 into new form factors and deployment models that our customers in 2006 could not have imagined. ## The full stack behind abundant intelligence DevFeed: [The full stack behind abundant intelligence]() Original publisher: [Read original article]() Published: 2026-08-25T07:05:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [OpenAI](), [Artificial Intelligence](), [Hardware](), [Low-Latency Inference](), [Inference Performance](), [model-deployment](), [gpt-oss](), [Benchmark](), [cloud-infrastructure](), [systems](), [Amazon Web Services](), [Microsoft]() Tags: [ai](), [aws](), [benchmark](), [company](), [compute](), [cost](), [efficiency](), [energy-efficiency](), [hardware](), [inference](), [low-latency-inference](), [microsoft](), [models](), [nvidia](), [openai]() ### AI overview OpenAI describes an integrated compute strategy spanning data centers, chips, models, software, products, and devices. It reports that its custom Jalapeño inference chip achieved higher peak throughput per kilowatt and lower token latency than commercial systems on the InferenceX benchmark using GPT-OSS 120B, while also performing strongly on DeepSeek R1 and Kimi K2. ### Source excerpt OpenAI CFO Sarah Friar explains how advances across chips, compute, models, and products compound to deliver more useful intelligence at greater scale and lower cost. ## Jalapeño's first results show industry-leading speed and efficiency in AI inference DevFeed: [Jalapeño's first results show industry-leading speed and efficiency in AI inference]() Original publisher: [Read original article]() Published: 2026-08-25T07:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [Inference](), [Inference Performance](), [OpenAI](), [Hardware](), [Benchmark](), [gpt-oss](), [deepseek]() Tags: [ai](), [benchmark](), [deepseek](), [efficiency](), [engineering](), [gpt-oss](), [hardware](), [inference](), [latency](), [models](), [openai](), [performance](), [software](), [speed](), [systems]() ### AI overview OpenAI reports initial results for Jalapeño, its custom inference chip. The article says the chip delivers higher throughput, lower end-to-end latency, and greater AI work per watt across GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 1T, based on tests using the InferenceX benchmark. ### Source excerpt Jalapeño is a custom inference chip from OpenAI that delivers faster, more power-efficient AI inference, with higher throughput and lower latency for modern models. ## With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents DevFeed: [With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents]() Original publisher: [Read original article]() Author: NVIDIA Writers Published: 2026-08-24T15:00:41Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Inference Performance](), [long-context](), [Low-Latency Inference](), [networking](), [gemma4]() Tags: [agentic](), [agents](), [ai-factory](), [ai-inference](), [ai-infrastructure](), [benchmark](), [bluefi](), [cloud](), [cpx](), [groq](), [groq-3-lpx](), [inference](), [infrastructure](), [latency](), [long-context](), [low-latency-inference](), [lpx](), [networking](), [nvidia](), [nvidia-spectrum-x-ethernet](), [nvidia-vera](), [nvidia-vera-rubin](), [nvl72](), [nvlink](), [vera-cpu](), [vera-rubin]() ### AI overview NVIDIA announces Groq 3 LPX production availability alongside Vera Rubin NVL72, positioning the integrated compute, networking, and inference stack for long-context, agentic AI workloads. The article highlights a benchmark result of 3,400 output tokens per second on Gemma 4 31B and describes partner adoption of Vera Rubin, Spectrum-X Multiplane, and Groq 3 LPX. ### Source excerpt The next era of AI inference won't be defined by a single breakthrough chip, network or system. It'll be defined by how every layer of the AI factory works together. That's why NVIDIA is extending Vera Rubin NVL72 with fast token generation for agentic systems. Announced today, the NVIDIA Vera Rubin rack-scale system NVIDIA Groq [...] ## Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents DevFeed: [Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents]() Original publisher: [Read original article]() Author: Shruti Koparkar Published: 2026-08-24T15:00:19Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [AI Agent](), [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [Inference Performance](), [long-context](), [GB300 NVL72](), [Inference](), [benchmarking](), [Nvidia]() Tags: [agentic-ai](), [agents](), [ai](), [ai-infrastructure](), [gb300-nvl72](), [hardware](), [inference](), [infrastructure](), [long-context](), [networking](), [nvidia-blackwell](), [nvidia-vera-rubin](), [performance](), [software](), [think-smart]() ### AI overview NVIDIA reports that Vera Rubin NVL72 systems deliver up to 30x higher throughput per megawatt than GB300 NVL72 on agentic workloads measured with the SemiAnalysis AgentX workload. The article attributes the efficiency challenge to long, variable agent workflows involving tool calls, accumulated context, and sub-agents. ### Source excerpt According to OpenRouter data, agentic AI workloads consume 15x more tokens than a simple chat request. Why? Consider what happens when an AI agent researches a company for an investment decision. The agent queries financial databases, searches news and filings, invokes a sub-agent to run peer comparisons and model valuations, then synthesizes everything into a [...] ## How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin DevFeed: [How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-08-24T15:00:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [d-matrix](), [Vera Rubin](), [Inference Performance](), [long-context](), [Benchmark](), [Artificial Intelligence](), [gemma4](), [systems](), [Cache]() Tags: [agent](), [agentic](), [agentic-ai-generative-ai](), [ai](), [ai-factory](), [ai-inference](), [benchmark](), [cache](), [collective](), [data-center-cloud](), [developer-tools-techniques](), [groq](), [groq-3-lpx](), [inference-performance](), [long-context](), [low-latency-inference](), [lpx](), [nvidia](), [nvidia-vera](), [performance](), [rubin-gpu](), [speed](), [systems](), [tokens](), [training-ai-models](), [vera-rubin](), [vera-rubin-nvl72]() ### AI overview NVIDIA Groq 3 LPX, paired with Vera Rubin NVL72, delivers high-interactivity AI inference for long-context workloads. A reported benchmark measured 3,431 output tokens per second on Gemma 4 31B with a 100K context. ### Source excerpt NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the... ## Open-weight models are fast on Neon AI Gateway. Here's why DevFeed: [Open-weight models are fast on Neon AI Gateway. Here's why]() Original publisher: [Read original article]() Author: Carlota Soto Published: 2026-08-20T12:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [databricks](), [Inference Performance](), [model-deployment](), [Low-Latency Inference](), [Caching](), [Mixture of Experts (MoE)](), [gpt-oss](), [Large Language Model]() Tags: [caching](), [databricks](), [gpt-oss](), [inference](), [mixture-of-experts-moe](), [optimization](), [performance](), [product](), [speed]() ### AI overview The article explains why open-weight models can deliver low latency on Neon AI Gateway when paired with Databricks Foundation Model APIs and an optimized serving stack. It highlights continuous batching, KV-cache paging, custom kernels, quantization, multi-GPU layouts, prompt caching, and Mixture-of-Experts architectures. ### Source excerpt Open-weight models are built for performance, but how they're served determines whether end users actually feel that speed. Neon AI Gateway calls Databricks Foundation Model APIs for Databricks-hosted open-weight models, a stack that has years of inference engineering behind it. [Next page]()