# vllm

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, providing an OpenAI-compatible API and support for diverse hardware.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## AMD ROCm 10.2 Brings hipFile to PyTorch, LVM, and mdraid Volumes, and Adds 18 Ryzen Processors

DevFeed: [AMD ROCm 10.2 Brings hipFile to PyTorch, LVM, and mdraid Volumes, and Adds 18 Ryzen Processors](<https://devfeed.tech/articles/amd-rocm-10-2-brings-hipfile-to-pytorch-lvm-and-mdraid-volumes-and-adds-18-ryzen-processors-146856.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/amd-rocm-10-2-hipfile-pytorch-lvm-mdraid-aiter-ryzen>)

Author: Harold Fritts

Published: 2026-11-20T16:18:56Z

Content type: release

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [rocm](<https://devfeed.tech/topics/rocm.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [LVM](<https://devfeed.tech/topics/lvm.md>), [Warp](<https://devfeed.tech/topics/warp.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [amd](<https://devfeed.tech/tags/amd.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [lvm](<https://devfeed.tech/tags/lvm.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [rocm](<https://devfeed.tech/tags/rocm.md>), [ryzen](<https://devfeed.tech/tags/ryzen.md>), [volumes](<https://devfeed.tech/tags/volumes.md>)

### AI overview

ROCm 10.2.0 extends AMD Infinity Storage and hipFile into PyTorch, fio, and ior, with support for LVM volumes and Linux mdraid sets. PyTorch's native `load_storage` and `save_storage` APIs move large tensors between GPU memory and storage; hipFile is integrated in fio 3.43 and ior's main branch for benchmarking that path. The stability-focused release also updates vLLM and SGLang, adds AITER support for Radeon and Ryzen AI, and supports 18 additional Ryzen processors. Other changes include expanded GPU libraries and an experimental unified `rocsys` profiler command. AMD's cited MLPerf and scaling figures are vendor-reported and were not independently audited; `rocsys` is experimental and Radeon AI Pro R9700 Hyperloom support is early access.

### Source excerpt

AMD has released ROCm 10.2.0, a stability-focused update to its open GPU software stack that refreshes the serving frameworks, libraries, and tools developers already run on Instinct, Radeon, and Ryzen hardware. For storage teams, hipFile and AMD Infinity Storage now plug into PyTorch, fio, and ior, and they work on volumes built with LVM and The post AMD ROCm 10.2 Brings hipFile to PyTorch, LVM, and mdraid Volumes, and Adds 18 Ryzen Processors appeared first on StorageReview.com.

## BlockJack: Cross-Tenant Redirection Fault Attacks on Paged KV Cache Address Translation in LLM Serving

DevFeed: [BlockJack: Cross-Tenant Redirection Fault Attacks on Paged KV Cache Address Translation in LLM Serving](<https://devfeed.tech/articles/blockjack-cross-tenant-redirection-fault-attacks-on-paged-kv-cache-address-translation-in-llm-serving-146051.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25945>)

Author: Qifan Wang, Jeremy Boy, Jonas Sander, Yansong Gao, Hongsheng Hu, Ye Dong, Thomas Eisenbarth, David Oswald

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [rowhammer](<https://devfeed.tech/topics/rowhammer.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [defense](<https://devfeed.tech/tags/defense.md>), [inference](<https://devfeed.tech/tags/inference.md>), [information](<https://devfeed.tech/tags/information.md>), [integrity](<https://devfeed.tech/tags/integrity.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [medical](<https://devfeed.tech/tags/medical.md>), [redirection](<https://devfeed.tech/tags/redirection.md>), [rowhammer](<https://devfeed.tech/tags/rowhammer.md>), [validation](<https://devfeed.tech/tags/validation.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

BlockJack corrupts vLLM-style paged KV cache block-table entries to redirect one tenant's cache reads to another tenant's blocks, without changing model parameters, prompts, or cached states. The authors report confidentiality and integrity attacks across seven Qwen, Llama, Gemma, and Phi models; on a 248-scene medical benchmark, up to 41.1% of responses recommended less urgent care while appearing consistent with the victim's question. They demonstrate physical feasibility using Rowhammer and evaluate assignment validation as a defense. vLLM and Sonar confirmed the vulnerability and proposed patches that the authors reviewed and validated.

### Source excerpt

arXiv:2611.25945v1 Announce Type: new Abstract: Large language model (LLM) serving frameworks accelerate inference by retaining previously computed states in a key-value (KV) cache. Paged serving frameworks such as vLLM organize this cache through block tables analogous to OS page tables, which map each request's logical KV blocks to physical KV blocks, forming a software-managed address translation layer. Although fault attacks such as Rowhammer have exploited OS page tables to redirect memory accesses, prior fault attacks on LLMs have largely targeted the data plane, i.e., model weights or KV-cache contents, leaving KV-cache mapping unexplored. We present BlockJack (i.e., Block Hijack), the first KV cache address-translation fault attack in LLM inference. By corrupting block-table entries, BlockJack redirects a request's KV cache reads to blocks assigned to another tenant's request without modifying model parameters, prompts, or cached KV states. It achieves both confidentiality violation (victim information appears in attacker responses) and integrity violation (attacker-supplied context steers victim answers). We study paged KV cache mappings in vLLM and formally characterize when redirect targets are predictable. Across seven models spanning the Qwen, Llama, Gemma, and Phi families, a faulted mapping exposes victim context. For the integrity attack, we examine how topic proximity, intra-topic divergence, and model behavior shape redirected responses. On a 248-scene medical benchmark, the integrity attack causes up to 41.1% of responses to recommend less urgent care, while still appearing consistent with the victim's question. We demonstrate physical feasibility of BlockJack with Rowhammer and subsequently evaluate assignment validation as a defense. Following responsible disclosure, vLLM and Sonar confirmed the vulnerability and proposed patches that we reviewed and validated.

## Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

DevFeed: [Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads](<https://devfeed.tech/articles/validation-driven-deployment-of-llm-inference-platforms-for-agentic-workloads-145608.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25905>)

Author: Shai Gretz, Alon Halfon, Yoav Katz, Lena Dankin, Michal Jacovi, Rachel Tzoref-Brill, Kfir Toledo, Elron Bandel, Natan Ergas, Michal Malka, Pravein Govindan Kannan, Maroon Ayoub, Michal Shmueli-Scheuer

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [code-coverage](<https://devfeed.tech/tags/code-coverage.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-d](<https://devfeed.tech/tags/llm-d.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traces](<https://devfeed.tech/tags/traces.md>), [validation](<https://devfeed.tech/tags/validation.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The paper presents an end-to-end validation pipeline for agentic LLM serving, from small-scale testing through production. In its experiments, replaying captured traces reproduced per-session input-token load more consistently than live reruns: at the 95th percentile, replay stayed within 1.18x of recorded tokens, while live reruns reached 4.98x. In the measured vLLM configuration, live, replayed, and synthetic workloads each exercised Python serving paths the others did not. A rollout case study of four large MoE models on llm-d found about half the optimal concurrency and 60% of the throughput on captured workloads compared with public traces. The authors say they open-source tools and release a trace dataset.

### Source excerpt

arXiv:2611.25905v1 Announce Type: new Abstract: Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.

## Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints

DevFeed: [Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints](<https://devfeed.tech/articles/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints-144471.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints/>)

Author: Hazim Qudah

Published: 2026-11-19T21:28:18Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [gpu monitoring](<https://devfeed.tech/topics/gpu-monitoring.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [amazon-sagemaker-studio](<https://devfeed.tech/tags/amazon-sagemaker-studio.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws-user-notifications](<https://devfeed.tech/tags/aws-user-notifications.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [drift](<https://devfeed.tech/tags/drift.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [gpu-utilization](<https://devfeed.tech/tags/gpu-utilization.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [latency](<https://devfeed.tech/tags/latency.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance](<https://devfeed.tech/tags/performance.md>), [serving](<https://devfeed.tech/tags/serving.md>), [studio](<https://devfeed.tech/tags/studio.md>)

### AI overview

Amazon SageMaker AI monitors eligible generative AI inference endpoints by default when they use vLLM serving containers and GPU instances. Using aggregated latency, GPU utilization, and serving configuration signals, it can flag week-over-week latency drift, mostly idle GPUs, or supported optimizations and updates that are missing. Monitoring does not read request or response content and has no GPU cost. Findings arrive through AWS Health and email; users can open them in SageMaker Studio and benchmark alternatives on separate compute, comparing response time, throughput, and cost per million tokens. No endpoint change occurs until the user chooses to apply a result, and keeping the current configuration is also an option. Benchmarks incur standard SageMaker charges or use an ML reservation. Monitoring can be disabled by setting OPT_OUT_TRACKING on a model container; doing so prevents telemetry collection and notifications for that endpoint.

### Source excerpt

Amazon SageMaker AI now monitors your generative AI inference endpoints by default and tells you when a change could improve performance or lower cost. It tracks latency, GPU utilization, and serving configuration, then sends a finding through AWS Health with a one-click path to benchmark alternatives in SageMaker Studio before you apply any change.

## GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits

DevFeed: [GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits](<https://devfeed.tech/articles/gatewatch-low-overhead-llm-monitoring-with-boolean-circuits-142814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24619>)

Author: Firas Spanioli, Yaniv David

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [boolean](<https://devfeed.tech/tags/boolean.md>), [circuits](<https://devfeed.tech/tags/circuits.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-monitoring](<https://devfeed.tech/tags/llm-monitoring.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [overhead](<https://devfeed.tech/tags/overhead.md>)

### AI overview

GateWatch represents each monitoring probe as Boolean logic over a few binarized sparse autoencoder (SAE) features. Its serving system computes only selected features, streams their binary values across layers, and evaluates classifiers in parallel with lookup tables. On 24 Gemma-2-9B tasks, it averages 80.49% test accuracy with 3.8 SAE features and 3.5 logic gates per probe--1.17 percentage points below full 16K-feature SAE probes--while matching or exceeding them on 10 tasks and reducing probe memory by 99.98%. In vLLM, 1,000 concurrent classifiers retain 97-98% of baseline throughput and 99.8% of KV-cache capacity.

### Source excerpt

arXiv:2611.24619v1 Announce Type: new Abstract: As large language models (LLMs) and LLM-based agents are increasingly deployed, monitoring their behavior becomes important for detecting unsafe or unintended actions. Activation probes offer an efficient approach, but existing methods face trade-offs between accuracy, interpretability, and computational overhead. Probes on raw activations lack interpretable inputs, whereas sparse autoencoder (SAE) probes use interpretable features but often rely on additive classifiers and large SAE encoders, limiting interpretability and increasing memory overhead. We introduce GateWatch, a compact, interpretable classifier that expresses its entire decision rule as an explicit Boolean logic over a small number of binarized SAE features. We explore three approaches to constructing these circuits under a specified gate budget. For efficient deployment, GateWatchServe computes only selected SAE features, enables streaming of their binary values across layers during inference, and evaluates multiple GateWatch classifiers in parallel using compact lookup tables. Across 24 tasks on Gemma-2-9B, GateWatch achieves 80.49% mean test accuracy using only 3.8 SAE features and 3.5 logic gates per probe on average, just 1.17 percentage points below full 16K-feature SAE probes, while matching or outperforming them on 10 tasks and reducing probe memory by 99.98%. Inside vLLM, 1,000 concurrent multi-layer, multi-token GateWatch classifiers retain 97-98% of baseline throughput and 99.8% of KV-cache capacity.

## Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference

DevFeed: [Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference](<https://devfeed.tech/articles/optimize-the-cob-then-the-kernel-optimizing-beam-search-for-generative-recommendation-inference-142637.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23339>)

Author: Shing Wai Pun, Kai Wu, Pengfei Xia, Hui Zang, Wei Guo, Yong Liu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ascend](<https://devfeed.tech/tags/ascend.md>), [attention](<https://devfeed.tech/tags/attention.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [beam-search](<https://devfeed.tech/tags/beam-search.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom](<https://devfeed.tech/tags/custom.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [engine](<https://devfeed.tech/tags/engine.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [graph](<https://devfeed.tech/tags/graph.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>)

### AI overview

Generative recommendation inference combines long prompts, wide beam searches, and few output tokens, which can make conventional LLM serving inefficient. The abstract says vLLM treats beams as independent sequences, repeating KV cache accesses over the shared prompt, and invokes the engine at each generation step, adding host-side processing and PagedAttention block-management overhead. The proposed Ascend NPU pipeline concatenates beams with Tree Attention and adds pre-generated partial-attention masks, static KV allocation, loop-unrolled dispatch, TorchAir graph compilation, operator fusion, and vocabulary-constrained decoding. Without custom kernels, it reports up to 25x lower end-to-end latency than vLLM and generation latency comparable to a state-of-the-art custom-kernel solution. These results are stated for Ascend NPUs.

### Source excerpt

arXiv:2611.23339v1 Announce Type: cross Abstract: Generative recommendation (GR) inference differs substantially from conventional LLM serving, featuring long input sequences, wide beam search, and only a few fixed output tokens. Existing LLM serving engines such as vLLM process beams as batched independent sequences, causing redundant KV cache accesses over the shared prompt prefix. Furthermore, vLLM executes each beam search generation step as a separate engine invocation, incurring host-side beam processing overhead between decoding steps and substantial runtime KV block management in PagedAttention. Using Tree Attention, we concatenate beams along the sequence dimension to eliminate redundant KV accesses, and further optimize execution through pre-generated partial-attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary-constrained decoding. Without the use of custom kernels, our approach reduces end-to-end latency by up to 25x over vLLM on Ascend NPUs, and achieves generation latency comparable to a SOTA solution using custom kernels, showing that the generation pipeline should be optimized holistically before resorting to custom kernel development.

## Language Model Security Modules: Turning LLM Interpretability into Runtime Defense

DevFeed: [Language Model Security Modules: Turning LLM Interpretability into Runtime Defense](<https://devfeed.tech/articles/language-model-security-modules-turning-llm-interpretability-into-runtime-defense-143106.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.25697>)

Author: XiuYu Zhang, Bonan Ruan, Junfeng Fang, An Zhang, Tat-Seng Chua, Zhenkai Liang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [model-security](<https://devfeed.tech/tags/model-security.md>), [runtime](<https://devfeed.tech/tags/runtime.md>)

### AI overview

LMSM adds a shared mediation path to LLM serving so interpretability signals can enforce runtime policies without each detector needing its own integration. Pluggable backends provide evidence such as probe or autoencoder signals; policies specify required evidence and evaluations, and LMSM verifies they ran before allowing output. In vLLM tests on Qwen3-4B and Gemma-3-4B, including eager and compiled execution, each evaluated backend retained at least 96% of native throughput with default optimizations. On Qwen3-4B, a representative probe policy reduced HarmBench attack success from 39.4% to 3.3%, while XSTest false refusals increased from 0.8% to 3.1%.

### Source excerpt

arXiv:2608.25697v2 Announce Type: replace Abstract: Large language models (LLMs) are served behind layered defenses such as alignment and guardrail models, yet jailbreaks still get through. Interpretability research looks inside the model, where methods such as probes and sparse autoencoders expose signals of harmful content during generation. A signal alone, however, is not a defense. To become one, the serving system must tie each signal to the right request, apply the policy as declared, and release output only when the policy allows. Today, defenses built on such signals handle these duties on their own, so every new detector needs its own integration. Yet none of these duties depends on how a signal is computed. We present Language Model Security Modules (LMSM), which adapts the design of Linux Security Modules to LLM serving and carries out these duties once in a shared mediation path. The path hosts pluggable backends that turn model internals into evidence and policies that decide what that evidence allows. Each policy declares the evidence and evaluations it needs, and LMSM provides exactly these for each request and verifies that they took place, so a failure is never mistaken for permission. Protection therefore remains cheap and correct under common serving optimizations. We evaluate LMSM in vLLM, with both eager and compiled execution, on Qwen3-4B and Gemma-3-4B with fitted probes, a released transcoder, and a released sparse autoencoder. With vLLM's default optimizations on, every backend keeps at least 96% of native throughput, and all of them share the same mediation path. On Qwen3-4B, a representative probe-based policy cuts HarmBench attack success from 39.4% to 3.3% while XSTest false refusals rise from 0.8% to 3.1%. LMSM lets existing and future interpretability methods become runtime defenses without rebuilding the serving path.

## CommitKV: Commit-Aligned KV Cache Compression for Multimodal Agents

DevFeed: [CommitKV: Commit-Aligned KV Cache Compression for Multimodal Agents](<https://devfeed.tech/articles/commitkv-commit-aligned-kv-cache-compression-for-multimodal-agents-143297.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.07855>)

Author: Weizhong Huang, Jinchao Zhang, Xiawu Zheng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-costs](<https://devfeed.tech/tags/inference-costs.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [memory](<https://devfeed.tech/tags/memory.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [multimodal-models](<https://devfeed.tech/tags/multimodal-models.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [redundancy](<https://devfeed.tech/tags/redundancy.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [times](<https://devfeed.tech/tags/times.md>)

### AI overview

CommitKV reduces multimodal agents' KV cache costs by comparing the deletion effect of the same page before a tool call completes and after its observation returns. It retires pages with low influence in both windows, temporarily protects pages influential before the call, then fills the cache budget using attention importance, key redundancy, and token-level deletion effects. Across four open-weight models and eight benchmarks, it achieved the highest average accuracy among the compared compression methods on each model. For Qwen3-VL-8B, accuracy was 69.85% at 512 tokens, 0.91% below FullKV; at 256 tokens, it beat the strongest compression baseline by 8.44%. In matched deployments with SGLang and vLLM at 256 tokens, peak allocated GPU memory fell 54.37%-55.23% and end-to-end inference was 2.12-2.17x faster than FullKV.

### Source excerpt

arXiv:2608.07855v2 Announce Type: replace Abstract: Multimodal agents gather visual evidence through reasoning and tool use, extending their ability to solve tasks beyond the initial input. However, accumulating observations increases KV cache memory and inference costs. Token eviction reduces these costs by retaining useful cached tokens. Existing methods use attention, redundancy, or stage-specific queries to rank tokens, without explicitly comparing the same history across tool interactions. When observations shift attention, they may mistake temporarily reduced influence for lost usefulness and evict information needed later. We propose CommitKV, a training-free KV cache compression method that accounts for this changing evidence use. Commit-aligned measurement compares the deletion effects of the same page before a tool call completes and after its observation returns. Lifecycle-aware compression retires pages with low influence in both windows and temporarily protects pages influential before the call. Joint scoring combines attention importance, key redundancy, and token-level deletion effects to fill the remaining budget. Extensive experiments cover four open-weight multimodal models and eight benchmarks. CommitKV achieves the highest average accuracy among the compared compression methods on all four models. On Qwen3-VL-8B, it achieves 69.85% at a 512-token budget, only 0.91% below FullKV, and exceeds the strongest compression baseline by 8.44% at 256 tokens. Deploying the Qwen3-VL-8B agent with SGLang and vLLM at a 256-token budget reduces peak allocated GPU memory by 54.37%-55.23% and accelerates end-to-end inference by 2.12-2.17$\times$ over FullKV on the matched workload.

## Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention

DevFeed: [Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention](<https://devfeed.tech/articles/efficient-wide-beam-generative-retrieval-with-model-native-trie-decoding-and-shared-context-attention-140011.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22309>)

Author: Vedant Jhaveri, Arashpreet Singh Mor, Satyam Kumar, Alex Tsun, Qing Lan, Shaunak Mishra, Sriram Vasudevan

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [beam](<https://devfeed.tech/tags/beam.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [generative](<https://devfeed.tech/tags/generative.md>), [model](<https://devfeed.tech/tags/model.md>), [native](<https://devfeed.tech/tags/native.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [trie](<https://devfeed.tech/tags/trie.md>), [wide](<https://devfeed.tech/tags/wide.md>)

### AI overview

Trie-Head reduces two serving costs in wide-beam generative retrieval: it stores the identifier trie as validated GPU tensors, scores compact codebook candidates, and streams the full-vocabulary normalizer, avoiding Python constraint callbacks and vocabulary-wide candidate buffers. The authors say it preserves valid-token scores in real arithmetic without retraining. Shared-context attention reuses prompt key/value blocks across beams while keeping decode tails private. Both mechanisms are implemented in SGLang and evaluated for candidate sourcing in LinkedIn Hiring Assistant and qualification retrieval on the Qualification Retrieval Platform. At beam width 200 on the latter workload, the combined system reports up to 5.57x closed-loop throughput over a Python custom logit processor baseline and up to 2.63x over BF16 FlashRec.

### Source excerpt

arXiv:2611.22309v1 Announce Type: new Abstract: Generative retrieval (GR) predicts catalog entries as short semantic identifiers (SIDs). Trie-constrained beam search keeps these identifiers valid, but wide beams expose two serving costs: vocabulary-wide score tensors that are mostly discarded, and repeated attention over the same prompt. We present Trie-Head, a model-native output interface that stores the identifier trie as validated GPU tensors. Compared with a Python custom logit processor (CLP), it reduces CPU--GPU coordination and memory traffic by avoiding Python constraint callbacks and vocabulary-wide candidate buffers. It scores compact codebook candidates and streams the full-vocabulary normalizer, preserving the original model's valid-token scores in real arithmetic without retraining. Shared-context attention complements this output path by reusing prompt key/value blocks across beams while preserving their private decode tails. We implement both mechanisms in SGLang and evaluate candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform. The combined system improves closed-loop throughput by up to 5.57x over the CLP baseline and up to 2.63x over BF16 FlashRec at a common beam width of 200 on the Qualification Retrieval Platform workload.

## SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding

DevFeed: [SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding](<https://devfeed.tech/articles/specsearch-towards-hardware-aware-efficient-speculative-decoding-140005.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22296>)

Author: Selin Yildirim, Ali Zeyrek, Deming Chen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [peft](<https://devfeed.tech/topics/peft.md>), [lora](<https://devfeed.tech/topics/lora.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [draft](<https://devfeed.tech/tags/draft.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [h100](<https://devfeed.tech/tags/h100.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [jev](<https://devfeed.tech/tags/jev.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [lora](<https://devfeed.tech/tags/lora.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-rtx](<https://devfeed.tech/tags/nvidia-rtx.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [precision](<https://devfeed.tech/tags/precision.md>), [sd](<https://devfeed.tech/tags/sd.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engine](<https://devfeed.tech/tags/search-engine.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

SpecSearch searches jointly over decoding strategy, draft length, and precision for each workload-GPU pair. Its analytical cost model accounts for memory and compute limits to narrow feasible configurations, and calibration measurements guide throughput optimization. A Jev-based predictor, trained by adapting frozen Qwen3-4B with LoRA adapters and a linear correction layer, selects configurations for held-out GPUs without deployment-GPU calibration. Across RTX 5090, H100, L40, and MI300X, fully calibrated SpecSearch averaged 2.01x autoregressive and 1.36x speculative-decoding throughput, with both baselines served by vLLM. The predictor achieved 1.30x throughput over vLLM speculative decoding and retained 97.36% of fully calibrated SpecSearch throughput.

### Source excerpt

arXiv:2611.22296v1 Announce Type: new Abstract: Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves $3.50\times$ the throughput of autoregressive (AR) decoding at low concurrency, but only $0.76\times$ at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of $2.01\times$ the throughput of AR and $1.36\times$ the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with $1.30\times$ throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains $97.36\%$ of the throughput achieved by SpecSearch that utilizes full calibration.

## Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels

DevFeed: [Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels](<https://devfeed.tech/articles/storing-the-gemma-4-k-v-cache-once-exact-global-kv-memory-savings-in-a-paged-serving-engine-with-unchanged-attention-kernels-140392.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23035>)

Author: Kushagra Bainsla

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [emulation](<https://devfeed.tech/tags/emulation.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [factor](<https://devfeed.tech/tags/factor.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [gemma-4-31b](<https://devfeed.tech/tags/gemma-4-31b.md>), [global](<https://devfeed.tech/tags/global.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [key](<https://devfeed.tech/tags/key.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [layers](<https://devfeed.tech/tags/layers.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nixl](<https://devfeed.tech/tags/nixl.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

For Gemma 4 31B global attention, the key is the value scaled by a fixed weight, so the authors store one 640-value row per token instead of separate 1,024-value key and value rows. An unchanged paged attention kernel reads the compact row through overlapping strided views. In a vLLM prototype, steady decode KV memory per request fell 1.34-1.40x at 32K-64K tokens; outputs matched an emulation of the same arithmetic, while 2 of 432 final answers differed from the stock engine within its run-to-run noise. With two A100 GPUs, NIXL transfer and CPU offload, a full 15 GiB decode pool handled 1.38x stock tokens per step for 16K prompts and 1.24x for 8K prompts; eager throughput gains were 1.30x and 1.11x. The prototype made equal-batch eager steps 4%-9% slower, and the method did not yet combine with an 8-bit KV cache at the default scale.

### Source excerpt

arXiv:2611.23035v1 Announce Type: cross Abstract: In the global attention layers of Gemma 4 31B, the value is the normalized key projection, and the key is the value scaled by a fixed weight, with rotary position encoding on one quarter of its dimensions. The Gemma 4 report states that this design reduces the global KV cache by up to 37.5%, but the open serving engines that we checked still store a full key and a full value: 1,024 values per KV head and token. We store one 640-value row instead, exact in real arithmetic, and an unchanged paged attention kernel reads it through two overlapping strided views as its key cache and its value cache. In a vLLM prototype with the 16-bit 31B model, the outputs are bit-identical to an emulation of the same arithmetic on the stock cache, 2 of 432 final answers differ from the stock engine (within its run-to-run noise), and the steady decode KV memory per request falls by a factor of 1.34 to 1.40 at 32K to 64K tokens. On a prefill/decode pair of two A100 GPUs with NIXL transfer and CPU offload, a decode engine with a full 15 GiB pool ran 1.38 times the stock decode tokens per step for 16K-token prompts (repeated to 0.06%) and 1.24 times for 8K-token prompts; a model of the vLLM block pool predicted both ratios to within 1% (the engine was 0.7% to 0.9% higher) and matched the change between them. The eager decode throughput rose by 1.30 and 1.11 times, and on this path the compact engine gave the same output tokens as the emulation for 8 prompts at batch 1. The run-time prototype makes an eager step 4% to 9% slower at equal batch, and the method does not yet stack with an 8-bit KV cache at the default scale.

## LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models

DevFeed: [LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models](<https://devfeed.tech/articles/loopzip-visual-token-compression-for-efficient-looped-vision-language-models-141803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22728>)

Author: Weizhong Huang, Zhicheng Liu, Yan Zhang, Fei Chao, Xiawu Zheng, Liujuan Cao

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

LoopZip is a training-free visual token compression method for looped vision-language models. It aggregates tokens into a compact candidate pool, selects tokens using averaged instruction attention from the first two calls of a shared attention layer, then applies shared value corrections fitted to both pre-selection outputs to reduce deviations across calls. The abstract reports tests on two open-source models, 11 benchmarks, and three pruning ratios. For LoopVL-1B, it retains 97.16% of original performance at 50% pruning and beats the strongest baseline by 5.36% at 87.5% pruning. Across Hugging Face Transformers, vLLM, and SGLang, reported peak allocated GPU memory drops by 58.36%-58.98%, with 2.01-2.47x end-to-end speedups over the original model.

### Source excerpt

arXiv:2611.22728v1 Announce Type: cross Abstract: Looped vision-language models (VLMs) deepen computation by reusing shared layers, approaching larger models' capabilities with fewer parameters and easing the storage and deployment burden of stacking independent layers. However, repeatedly processing long visual sequences still incurs substantial compute and GPU memory costs. Visual token compression reduces these costs by shortening sequences. Existing methods mainly select original tokens directly or combine selection with aggregation. In looped models, the former may overlook changing token importance across calls and discard information needed later; the latter lack joint consideration of output changes across calls, limiting information preservation. We propose LoopZip, a training-free visual token compression algorithm addressing both issues. Visual aggregation builds a compact candidate pool before the looped backbone, and cross-invocation selection averages instruction attention from the first two existing calls of the same shared attention layer to retain tokens needed by both states. Weight-tied value correction then jointly fits shared visual-value adjustments to both pre-selection layer outputs, reducing output deviations across states. To validate LoopZip's effectiveness, we conduct extensive experiments covering two open-source looped VLMs, eleven benchmarks, and three pruning ratios. On LoopVL-1B, LoopZip retains 97.16% of the original model's performance at 50% pruning ratio and exceeds the strongest baseline by 5.36% at 87.5% pruning ratio. Across Hugging Face Transformers, vLLM, and SGLang, LoopVL-1B with LoopZip reduces peak allocated GPU memory by 58.36%-58.98% and achieves 2.01-2.47x end-to-end speedups over the original model.

## Communicate the Change: Efficient Expert Parallelism via Activation Compression

DevFeed: [Communicate the Change: Efficient Expert Parallelism via Activation Compression](<https://devfeed.tech/articles/communicate-the-change-efficient-expert-parallelism-via-activation-compression-139983.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22215>)

Author: Fengyuan Liu, Jay Gala, Siva Reddy, Dzmitry Bahdanau

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-inference](<https://devfeed.tech/tags/distributed-inference.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [ethernet](<https://devfeed.tech/tags/ethernet.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>)

### AI overview

EchoMoE reduces communication overhead in expert-parallel MoE inference by sending a compressed difference between the current hidden state and a periodically refreshed earlier-layer reference, rather than sending each hidden state in full. The abstract reports 58% lower inter-node communication volume and 22.6% higher throughput, with average performance within 3.8 percentage points across 18 benchmarks. On bandwidth-constrained Ethernet, throughput improves by up to 2.38x. The modification is designed to retrofit pretrained MoE models, requiring about 8,500x fewer training tokens than full pretraining; the communicated representations can also be quantized without measurable quality loss. The authors report an implementation for pretrained models and an inference integration for vLLM.

### Source excerpt

arXiv:2611.22215v1 Announce Type: cross Abstract: Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.

## Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell

DevFeed: [Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell](<https://devfeed.tech/articles/beyond-bitwidth-deployment-constrained-mixed-precision-quantization-for-multilingual-llm-inference-on-nvidia-blackwell-141712.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21974>)

Author: Hyojung Han

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [english](<https://devfeed.tech/tags/english.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gptq](<https://devfeed.tech/tags/gptq.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [int4](<https://devfeed.tech/tags/int4.md>), [int8](<https://devfeed.tech/tags/int8.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [multilingual-llm](<https://devfeed.tech/tags/multilingual-llm.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-blackwell](<https://devfeed.tech/tags/nvidia-blackwell.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

On Qwen3-8B running with vLLM on an NVIDIA B200, the tested mixed-precision setup met the study's preregistered quality, memory, and latency gates: 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head used 5.917 GiB against a 5.98 GiB limit, retained 0.904 Korean teacher-correct accuracy against a 0.90 gate, and reached a 0.939 TPOT ratio relative to BF16. On the untouched 5,168-item FINAL-81 set, it improved paired accuracy over NVFP4 W4A4 by 1.4 percentage points (adjusted interval +0.2 to +2.6), less than its 3.4-point selection-set gain. Packed INT2 and INT3 executed, but all seven INT2 grids degenerated during generation. Restoration effects were non-additive, and Korean and reasoning sensitivity rankings differed. Because the comparison changes both activation precision and INT8 protection, the isolated allocation effect is unclear. Final checkpoints were not retained, so bitwise rebuild reproducibility is unverified.

### Source excerpt

arXiv:2611.21974v1 Announce Type: cross Abstract: Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.

## Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM

DevFeed: [Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM](<https://devfeed.tech/articles/keeping-agents-fast-high-interactivity-decode-with-flydsl-monokernels-in-vllm-140940.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-11-18-flydsl-monokernels>)

Author: Felix Li, Lingpeng Jin, Hattie Wu, Carlus Huang, Fangzhou Ai, Zaidy Ali, Amanzhol Salykov, Hyukjoon Lee, Ziming Huang, Jun Chen, Yifeng Huang, Bin Ding, Xiaobing Zhang, Yi Gan, Vinayak Gokhale, Hongxi

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [rocm](<https://devfeed.tech/topics/rocm.md>), [mlir](<https://devfeed.tech/topics/mlir.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [amd](<https://devfeed.tech/tags/amd.md>), [decode](<https://devfeed.tech/tags/decode.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [handoffs](<https://devfeed.tech/tags/handoffs.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [integration](<https://devfeed.tech/tags/integration.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model](<https://devfeed.tech/tags/model.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

FlyDSL MonoKernels aim to reduce low-concurrency decode latency in vLLM on AMD Instinct GPUs by combining dependent stages into a few persistent kernel launches. Authors choose model-specific boundaries and schedules; FlyDSL compiles them, while vLLM selects the opt-in path for supported steps. Thread blocks can reuse work across stages, prefetch weights, and hand off ready tiles through tagged mailboxes, though dependencies and memory traffic remain. Reported evaluations show gains across several models, with results varying by model, GPU, concurrency, and benchmark; some comparisons use historical baselines. The paths preserve vLLM's scheduler and KV cache, fall back to stock layers for unsupported workloads, and require accuracy checks per configuration. Limits include reduced gains at higher concurrency, maximum step widths, and extra weight copies on MI300X/MI325X.

### Source excerpt

How FlyDSL MonoKernels keep agents responsive with high-interactivity decode in vLLM on AMD Instinct GPUs, using model-specific fusion, data-ready handoffs, and specialized-decode integration.

## Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

DevFeed: [Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference](<https://devfeed.tech/articles/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-138699.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference/>)

Author: Vinay Arora

Published: 2026-11-17T23:33:26Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon SageMaker HyperPod](<https://devfeed.tech/topics/amazon-sagemaker-hyperpod.md>), [Dynamo](<https://devfeed.tech/topics/dynamo.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agent-skills](<https://devfeed.tech/tags/agent-skills.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-hyperpod](<https://devfeed.tech/tags/amazon-sagemaker-hyperpod.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [api](<https://devfeed.tech/tags/api.md>), [aws](<https://devfeed.tech/tags/aws.md>), [benchmark-performance](<https://devfeed.tech/tags/benchmark-performance.md>), [disaggregated-serving](<https://devfeed.tech/tags/disaggregated-serving.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-inference](<https://devfeed.tech/tags/distributed-inference.md>), [dynamo](<https://devfeed.tech/tags/dynamo.md>), [inference](<https://devfeed.tech/tags/inference.md>), [managed](<https://devfeed.tech/tags/managed.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-dynamo](<https://devfeed.tech/tags/nvidia-dynamo.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [routing](<https://devfeed.tech/tags/routing.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

SageMaker HyperPod adds managed NVIDIA Dynamo deployments for large models that may exceed one instance's GPU memory or benefit from separating prompt processing and token generation. Users can bring an existing DynamoGraphDeployment or configure an InferenceEndpointConfig with dynamoSpec; the latter supports vLLM by default, plus SGLang and TensorRT-LLM. Dynamo provides multi-node and aggregated or disaggregated serving, while HyperPod components coordinate GPU placement, EFA networking, node health, and recovery. In benchmarks described in the source, disaggregated Qwen3-235B serving reached 207 versus 101 output tokens per second per GPU at selected SLA-compliant points. A two-node Qwen3.5-397B aggregated setup retained 99.1% of single-node throughput under the tested workload. The article advises selecting a matching recipe, verifying EFA transport, benchmarking representative traffic against SLAs, and tuning based on measured bottlenecks; results depend on the tested models and workloads.

### Source excerpt

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed inference, with multi-node and disaggregated serving, KV-aware routing, and automatic fault recovery. Serve large models across many nodes without building the orchestration layer yourself.

## Three questions to ask about AI inference in Amazon EKS, and how Amazon CloudWatch answers them

DevFeed: [Three questions to ask about AI inference in Amazon EKS, and how Amazon CloudWatch answers them](<https://devfeed.tech/articles/three-questions-to-ask-about-ai-inference-in-amazon-eks-and-how-amazon-cloudwatch-answers-them-139426.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/mt/three-questions-to-ask-about-ai-inference-in-amazon-eks-and-how-amazon-cloudwatch-answers-them/>)

Author: Jean Velez Torres

Published: 2026-11-17T18:52:53Z

Content type: tutorial

Language: en

Sources: [AWS Cloud Operations Blog](<https://devfeed.tech/sources/aws-cloud-operations-blog.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [alarm](<https://devfeed.tech/tags/alarm.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [cloudwatch-omni](<https://devfeed.tech/tags/cloudwatch-omni.md>), [costs](<https://devfeed.tech/tags/costs.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The Amazon CloudWatch Observability add-on for Amazon EKS joins DCGM GPU metrics with Kubernetes pod and namespace labels, letting teams attribute GPU use without a manual join. The post shows how accelerated compute insights, CloudWatch Omni, and PromQL queries can reveal GPU ownership and idle capacity, compare hardware activity with inference throughput and KV cache headroom, and associate health warnings or XID errors with workloads. In its test cluster, two A10G GPUs stayed allocated during nearly nine hours without requests: utilization fell to 0%, but vLLM-held memory stayed near 21,300 MiB per GPU. A model comparison found similar throughput and cost per token, while Qwen used more host CPU and memory and held nearly twice as many KV-cache tokens as Mistral; higher-load limits were not tested. The article also explains add-on configuration, PromQL alarms, and logging costs. Findings are limited to two g5.2xlarge instances with one GPU each; multi-node inference and other instance families were not tested.

### Source excerpt

GPUs are expensive and scarce. On a cluster shared by several AI teams, an idle GPU costs you twice: once on the bill, and again as capacity a team is waiting for. Three questions follow: which workload holds each GPU, how well it uses it, and which workloads are affected when one GPU goes bad. [...]

## Sarvam Launches Sarvam Token Factory: Bringing India-Hosted Inference to 8 Open Models

DevFeed: [Sarvam Launches Sarvam Token Factory: Bringing India-Hosted Inference to 8 Open Models](<https://devfeed.tech/articles/sarvam-launches-sarvam-token-factory-bringing-india-hosted-inference-to-8-open-models-137112.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/11/17/sarvam-token-factory-india-inference-platform/>)

Author: Asif Razzaq

Published: 2026-11-17T09:51:18Z

Content type: news

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Dynamo](<https://devfeed.tech/topics/dynamo.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [gemma](<https://devfeed.tech/topics/gemma.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [factory](<https://devfeed.tech/tags/factory.md>), [for-devs](<https://devfeed.tech/tags/for-devs.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [gemma-4-31b](<https://devfeed.tech/tags/gemma-4-31b.md>), [glm](<https://devfeed.tech/tags/glm.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-clusters](<https://devfeed.tech/tags/gpu-clusters.md>), [india](<https://devfeed.tech/tags/india.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [staff](<https://devfeed.tech/tags/staff.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [token](<https://devfeed.tech/tags/token.md>), [token-factory](<https://devfeed.tech/tags/token-factory.md>)

### AI overview

Sarvam Token Factory serves Sarvam and open-weight models from GPU clusters in India through one OpenAI-compatible API; enterprises can also request on-premise deployment. The launch lists eight models, with shared endpoints for pay-as-you-go access and provisioned throughput required for some models. Its stack separates prefill and decode GPU pools and uses cache-aware routing and a shared cache. Sarvam reports performance gains from caching and other optimizations, including fitting Sarvam 105B on one B200 per replica. Its self-reported benchmarks show advantages in several tested cases, such as Kimi K3.1 Pro throughput versus InferenceX, but Together reports faster first-token latency and streaming on DeepSeek V4.1 Flash. Introductory prices run through 2026; provisioned model rates are not published in the excerpt.

### Source excerpt

Sarvam Token Factory is now open to everyone. It serves Sarvam 105B, GLM 5.3, DeepSeek V4.1 Flash, Kimi K3.1 Pro and Gemma 4 31B from GPU clusters in India through one OpenAI-compatible API. The post Sarvam Launches Sarvam Token Factory: Bringing India-Hosted Inference to 8 Open Models appeared first on MarkTechPost.

## Where the Savings Go: The Limits of Hiding Cross-Tenant Cache Hits in LLM Serving

DevFeed: [Where the Savings Go: The Limits of Hiding Cross-Tenant Cache Hits in LLM Serving](<https://devfeed.tech/articles/where-the-savings-go-the-limits-of-hiding-cross-tenant-cache-hits-in-llm-serving-139296.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20291>)

Author: Bowen Sun, Chaowei Xiao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [random](<https://devfeed.tech/tags/random.md>), [request](<https://devfeed.tech/tags/request.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [share](<https://devfeed.tech/tags/share.md>), [trace](<https://devfeed.tech/tags/trace.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

Sharing an LLM prefix cache can leak whether another tenant used a text through saved computation that speeds later queued requests. In step-based schedulers that keep working while requests wait, the saved time shifts every queued request earlier. A rate-limited attacker can bracket a probe with two groups of one-token listeners and compare their completion steps. In a replayed production trace on one vLLM instance at up to half capacity, one probe detected cached texts of 256 tokens or more with advantage at least 0.98. Hiding only the hit request still allowed detection with one probe on vLLM and SGLang; random delays had to reach roughly ten times the shift or more to reduce advantage to 0.1. The abstract also bounds a tenant's gain from content written by k other accounts by order k times the permitted leakage: with zero leakage, only the provider retains the savings. Exact hiding is possible in principle on one instance, but pause-or-hold mechanisms under saturated prefill must pad every step that might hold a hit, with cost growing over time rather than with hits.

### Source excerpt

arXiv:2611.20291v1 Announce Type: new Abstract: LLM serving engines cache prompt prefixes; a cache hit skips computation and returns faster, so a shared cache reveals whether others have sent a given text. Per-account salting closes this channel but forgoes cross-tenant reuse and isolates only whole accounts, even where access is redistributed or trust domains are subdivided below them. Defenses that share globally but hide hits promise reuse without relying on account boundaries. Hiding a hit requires everything tenants observe afterwards to be distributed as if it had missed. The defenses we examine target the hit-miss gap; we instead follow where the saved time goes. Step-based schedulers that never idle while work waits pass it in full to every request queued behind the hit, shifting each earlier by the same amount. An attacker within ordinary rate limits brackets a probe with two groups of listeners that each return one token, sends them in one short burst, and counts in how many steps they finish. Replaying a production trace on one vLLM instance at up to half capacity, a single probe detects cached texts of 256 tokens or more with advantage at least 0.98. Hiding only the hit request falls to one probe on vLLM and SGLang; random delays must reach about ten times the shift or more to cut the advantage to 0.1; narrowing what is shared gives no guarantee against this inference. Since any observable tenant benefit is also a signal, we prove that a rate-limited tenant's gain from content written by k other accounts is at most of order k times the leakage allowed, whatever the provider does with identities; at zero leakage, only the provider keeps the savings. Exact hiding is attainable in principle on one instance, but mechanisms that only pause or hold work know skipped work's duration only approximately, so under saturated prefill they must pad every step that may hold a hit, at a price that grows with time rather than with hits.

## High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion

DevFeed: [High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion](<https://devfeed.tech/articles/high-throughput-serving-of-nemotron-3-5-transcribe-with-regime-aware-megakernels-and-kernel-fusion-136987.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20095>)

Author: Aditya Chatterjee, Sanjay Singh Chauhan, Mayank Jain, Arushi Dhingra, Sahin Chowdhury, Rahul Mittal

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asr](<https://devfeed.tech/tags/asr.md>), [batching](<https://devfeed.tech/tags/batching.md>), [beats](<https://devfeed.tech/tags/beats.md>), [best](<https://devfeed.tech/tags/best.md>), [budget](<https://devfeed.tech/tags/budget.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [completion](<https://devfeed.tech/tags/completion.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [eess-as](<https://devfeed.tech/tags/eess-as.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [megakernels](<https://devfeed.tech/tags/megakernels.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nemotron-3-5](<https://devfeed.tech/tags/nemotron-3-5.md>), [nemotron-3-5-transcribe](<https://devfeed.tech/tags/nemotron-3-5-transcribe.md>), [serving](<https://devfeed.tech/tags/serving.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The serving engine adapts kernel fusion per inference pass, choosing between persistent megakernels and fused-kernel chains using offline-measured crossover points as continuous batching changes the workload. On H100 SXM it reaches 4,345x real-time throughput (1.39x tuned vLLM at peak, and up to 4.39x faster at low concurrency); on B200 it reaches 6,367x (1.64x vLLM at the highest measured concurrency). WER is 1.91-1.93% versus vLLM's 1.88-1.93%. Within a 400 ms p95 completion-time budget, it serves eight times as many concurrent requests. Per-pass selection outperforms always using megakernels by 23% at 128 concurrent requests and always using fused kernels by 9% at eight. Opt-in options, including greedy MoE layer skipping, add 13-16% throughput on B200 for offline jobs that accept higher WER.

### Source excerpt

arXiv:2611.20095v1 Announce Type: cross Abstract: Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR leaderboards. Serving it efficiently takes more than fast operators: continuous batching changes the amount and mix of work at every iteration, and the best fusion granularity changes with it. We present a serving engine that, for every pass, chooses between persistent megakernels and chains of fused kernels at crossovers measured offline. On an H100 SXM GPU, throughput of our accuracy-preserving serving peaks at 4,345x real time, 1.39x the peak of tuned vLLM, and is up to 4.39x faster at low concurrency; on B200 it peaks at 6,367x, 1.64x vLLM at the highest concurrency we measured. Word error rate (WER) is 1.91 to 1.93% against 1.88 to 1.93% for vLLM. Within a 400 ms budget for p95 end-to-end completion time, it serves 8x as many concurrent requests. Choosing per pass beats either level alone: 23% higher throughput than always using megakernels at 128 concurrent requests, and 9% higher than always using fused kernels at 8. Opt-in switches, including greedy MoE layer skipping, add 13 to 16% throughput on B200 for offline jobs that can accept higher WER.

## Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

DevFeed: [Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters](<https://devfeed.tech/articles/benchmarking-llm-inference-on-desktop-scale-nvidia-grace-blackwell-clusters-136990.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20289>)

Author: Kevin Bhimani, Christian Escobar, Daniel Mihovch, Rudolf Eigenmann, Sunita Chandrasekaran

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [DGX Spark](<https://devfeed.tech/topics/dgx-spark.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [desktop](<https://devfeed.tech/tags/desktop.md>), [dgx-spark](<https://devfeed.tech/tags/dgx-spark.md>), [energy](<https://devfeed.tech/tags/energy.md>), [gb10](<https://devfeed.tech/tags/gb10.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [grace-blackwell](<https://devfeed.tech/tags/grace-blackwell.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>)

### AI overview

Benchmarks of 17 open-weight models on one or two RDMA-connected NVIDIA DGX Spark GB10 systems find that deployment strategy changes the benefit of adding a second system. With vLLM, tensor parallelism (TP=2) delivered a median 1.59x peak-throughput gain over one Spark, compared with 1.39x for pipeline parallelism (PP=2); TP=2 also reduced unloaded inter-token latency to 0.52-0.73x of one Spark, while PP=2 offered little reduction. For models that fit on one Spark, TP=2 generally suited low concurrency, whereas two replicas achieved higher throughput at high concurrency. One Spark used less GPU energy per output token at matched concurrency. The abstract also reports that NVFP4 reduced unloaded latency more than adding a second Spark at BF16, and suggests collective communication overhead helps explain TP=2's advantage.

### Source excerpt

arXiv:2611.20289v1 Announce Type: new Abstract: Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.

## FlashSampling: Fast and Memory-Efficient Exact Sampling

DevFeed: [FlashSampling: Fast and Memory-Efficient Exact Sampling](<https://devfeed.tech/articles/flashsampling-fast-and-memory-efficient-exact-sampling-134969.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.15854>)

Author: Tomas Ruiz, Zhen Qin, Yifan Zhang, Xuyang Shen, Yiran Zhong, Mengdi Wang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [memory](<https://devfeed.tech/tags/memory.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

FlashSampling fuses exact categorical sampling into the language-model head matrix multiplication, avoiding materialization of the logits tensor in HBM. It processes logits in on-chip tiles, adds Gumbel noise, retains each tile's maximizer, then reduces across tiles. For tensor-parallel decoding, it replaces logits all-gather with streaming peer-to-peer writes that overlap communication with computation and memory loads across up to eight GPUs; the abstract reports near-ideal scaling at large batch sizes. The method's exactness relies on argmax decomposition and hierarchical factorization for grouped variants. Kernel speedups are reported across H100, H200, B200, and B300 GPUs; end-to-end vLLM tests reduce time per output token by up to 10% on the tested models.

### Source excerpt

arXiv:2603.15854v4 Announce Type: replace-cross Abstract: Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We present FlashSampling, an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in HBM. The method is simple: compute logits tile-by-tile on chip, add Gumbel noise, keep only one maximizer per row and per vocabulary tile, and finish with a small reduction over tiles. In tensor-parallel decoding, FlashSampling replaces the all-gather of logits with streaming peer-to-peer writes: This overlaps GPU-to-GPU communication with computation and HBM loads across up to 8 GPUs, with near-ideal scaling at large batch sizes. Our kernel is exact because argmax decomposes over partitions; grouped variants for online and tensor-parallel settings are exact by hierarchical factorization of the categorical distribution. FlashSampling demonstrates kernel-level speedups on decode workloads across 4 different datacenter GPUs (H100, H200, B200, B300), and in end-to-end vLLM experiments, it reduces time per output token by up to $10%$ on the models we test. These results show that exact sampling, with no approximation, can be integrated into the matmul itself, consolidating the bandwidth-bound sampling step in an efficient epilogue.

## Compiling Specialized GPU-Initiated Networking

DevFeed: [Compiling Specialized GPU-Initiated Networking](<https://devfeed.tech/articles/compiling-specialized-gpu-initiated-networking-134896.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18145>)

Author: Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [device-code](<https://devfeed.tech/tags/device-code.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gin](<https://devfeed.tech/tags/gin.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [h100](<https://devfeed.tech/tags/h100.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [libraries](<https://devfeed.tech/tags/libraries.md>), [lifetimes](<https://devfeed.tech/tags/lifetimes.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [networking](<https://devfeed.tech/tags/networking.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvshmem](<https://devfeed.tech/tags/nvshmem.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>)

### AI overview

Obaleth integrates GPU-initiated communication into a kernel's intermediate representation so a compiler can co-optimize communication with computation and generate specialized transport code. It schedules completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. The abstract reports that on H100s it sends small messages up to 3.3x faster; on 256 GPUs its all-to-all runs 1.25x and 2.3x faster than NVSHMEM and NCCL GIN, respectively, with lower memory and register use. In vLLM, serving Qwen3-235B on 16 B200 GPUs, it answers requests up to 1.29x faster than DeepEP.

### Source excerpt

arXiv:2611.18145v1 Announce Type: new Abstract: Machine learning systems specialize computation to the workload and hardware, but communication remains in generic libraries whose device code and transport decisions are not visible to high-level compilers. We present Obaleth, a compiler that makes GPU-initiated communication part of a kernel's intermediate representation, co-optimizes it alongside computation, and lowers it to specialized transport code. Obaleth plans completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. On H100 GPUs, Obaleth sends small messages up to 3.3$\times$ faster, and on 256 GPUs runs all-to-all 1.25$\times$ and 2.3$\times$ faster than NVSHMEM and NCCL GIN, with much smaller memory and register use. In vLLM, it serves Qwen3-235B on 16 B200 GPUs and answers requests up to 1.29$\times$ faster than with DeepEP.

## When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

DevFeed: [When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM](<https://devfeed.tech/articles/when-one-gpu-isn-t-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm-132447.md>)

Original publisher: [Read original article](<https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/>)

Author: Fatima Fall

Published: 2026-11-13T17:00:00Z

Content type: tutorial

Language: en

Sources: [NVIDIA Developer](<https://devfeed.tech/sources/nvidia-developer.md>), [NVIDIA Technical Blog](<https://devfeed.tech/sources/nvidia-technical-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [DGX Station](<https://devfeed.tech/topics/dgx-station.md>), [gpudirect rdma](<https://devfeed.tech/topics/gpudirect-rdma.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-ai-generative-ai](<https://devfeed.tech/tags/agentic-ai-generative-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [dgx](<https://devfeed.tech/tags/dgx.md>), [dgx-station](<https://devfeed.tech/tags/dgx-station.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [llm-benchmarking](<https://devfeed.tech/tags/llm-benchmarking.md>), [llms](<https://devfeed.tech/tags/llms.md>), [local-ai](<https://devfeed.tech/tags/local-ai.md>), [networking-communications](<https://devfeed.tech/tags/networking-communications.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [serving](<https://devfeed.tech/tags/serving.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

The tutorial runs the 753-billion-parameter GLM-5.3 NVFP4 model across two GB300 DGX Stations because its roughly 464 GB checkpoint exceeds one GPU's 252 GB of memory. vLLM tensor parallelism splits weights and computation across both GPUs; ConnectX-8 RDMA carries rank-to-rank traffic, and Station A exposes a single OpenAI-compatible API. The workflow covers validating the link, downloading and syncing identical checkpoint copies, launching one vLLM rank per station, checking RDMA and API health, and measuring load with AIPerf on 80 SPEED-Bench coding conversations. Prerequisites include substantial disk space on each station, SSH access, Hugging Face access to the gated dataset, and acceptance of the model and dataset terms. Reported throughput peaks at 607.8 tokens/s at concurrency 16, then drops to 385.0 tokens/s at concurrency 32, where mean latency is 453.5 seconds; mean per-user throughput declines as concurrency rises. The excerpt notes that longer prompts or more concurrent users need more KV cache, and suggests reducing context length or request concurrency if memory runs out.

### Source excerpt

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...

[Next page](<https://devfeed.tech/topics/vllm.md?cursor=WyIyMDI2LTExLTEzVDE3OjAwOjAwKzAwOjAwIiwgIjIzMDFlZjExLTYwMTktNGUwMy1iOThiLWU0YzllODA0YzU3OCJd>)