# vllm vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, providing an OpenAI-compatible API and support for diverse hardware. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut DevFeed: [NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut]() Original publisher: [Read original article]() Author: Zhihan Jiang Published: 2026-09-16T15:00:48Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [benchmarking](), [Inference Performance](), [Dynamo](), [vllm](), [Nvidia](), [TensorRT-LLM]() Tags: [ai-infrastructure](), [benchmarks](), [dynamo](), [efficiency](), [hardware](), [inference](), [mlperf](), [networking](), [nvidia](), [nvidia-vera-rubin](), [nvl72](), [optimization](), [performance](), [software](), [tensorrt](), [tensorrt-llm](), [vera-rubin-nvl72](), [vllm]() ### AI overview NVIDIA reports MLPerf Inference v6.1 preview results for Vera Rubin NVL72 and GB300 NVL72 systems. Vera Rubin NVL72 delivered up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x higher throughput on DeepSeek-R1, while a four-rack GB300 NVL72 submission achieved 99% scaling efficiency. The results used vLLM, NVIDIA Dynamo, and TensorRT-LLM. ### Source excerpt System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. [...] ## Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU DevFeed: [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU]() Original publisher: [Read original article]() Author: Anthony Su; Injae Kwak Published: 2026-09-12T11:04:33.891311Z Content type: article Language: en Sources: [Google Developers Blog]() Topics: [Embeddings](), [vllm](), [Inference](), [long-context](), [autoscaling](), [Kubernetes](), [Cloud](), [Google](), [Artificial Intelligence](), [model-deployment](), [Open Source]() Tags: [autoscaling](), [embedding](), [github](), [gpu](), [inference](), [kubernetes](), [long-context](), [multimodal](), [open-source](), [retrieval](), [tpu](), [vllm]() ### AI overview This article explains how Google Cloud integrated native TPU support into vLLM to serve long-context, multimodal embedding models at production scale. It describes elastic TPU provisioning with Google Kubernetes Engine, autoscaling across accelerator types, and TPU-specific optimizations for long sequences and chunked prefill. The resulting setup is designed to preserve numerical parity with GPU reference baselines while supporting high-throughput semantic retrieval applications. ### Source excerpt Google Cloud has natively integrated TPU support into the vLLM serving engine, allowing developers to elastically scale high-demand embedding pipelines using Google Kubernetes Engine (GKE). To handle massive 15K+ token contexts for models like Qwen3-Embedding-8B, the engineering team implemented TPU-specific optimizations such as hardware-safe tensor alignment, JAX/XLA compilation pre-warming, and a hybrid StepPool architecture for chunked prefill management. These enhancements achieve near-perfect numerical parity with reference GPU baselines, and developers can immediately leverage the open-sourced setup recipes on the AI-Hypercomputer GitHub to build their own high-throughput semantic retrieval applications. ## The Architecture for Serving 100 Fine-Tuned Models on One GPU DevFeed: [The Architecture for Serving 100 Fine-Tuned Models on One GPU]() Original publisher: [Read original article]() Author: Avi Chawla Published: 2026-09-11T21:25:15Z Content type: tutorial Language: en Sources: [Daily Dose of Data Science]() Topics: [GPU](), [vllm](), [Fine-tuning](), [lora](), [Serverless](), [Deployment](), [Code]() Tags: [code](), [deployment](), [fine-tuning](), [gpu](), [lora](), [memory](), [models](), [production](), [serverless](), [vllm](), [workers]() ### AI overview This tutorial compares architectures for serving 100 fine-tuned 7B model variants on GPUs. It explains how separate merged models increase storage, GPU memory use, scaling pools, cold starts, and idle capacity, while a shared base model with LoRA adapters enables adapter reuse through vLLM. The article plans to test merged, unmerged startup-loaded, request-time adapter loading, and hosted-per-tenant deployments on Runpod Serverless. ### Source excerpt ...explained with code. ## Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL DevFeed: [Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Sergio Paniego Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [lora](), [grpo](), [jobs](), [vllm](), [NCCL](), [async](), [hugging face]() Tags: [async](), [grpo](), [hub](), [hugging-face](), [jobs](), [llm](), [lora](), [nccl](), [open-source](), [rl](), [storage](), [trl](), [vllm]() ### AI overview This article describes asynchronous GRPO training with a LoRA adapter across separate Hugging Face Jobs. The adapter is synchronized to vLLM replicas through a shared Storage Bucket, while a proxy handles authentication, rollout routing, and adapter-load broadcasts. Five runs reduced the time for 500 steps from 3 hours 27 minutes to 53 minutes. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Optimize vLLM speculative decoding with FastMTP heads DevFeed: [Optimize vLLM speculative decoding with FastMTP heads]() Original publisher: [Read original article]() Author: Rahul Tuli Published: 2026-09-08T14:20:16Z Content type: article Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [vllm](), [Inference](), [Large Language Model](), [Fine-tuning](), [model-deployment](), [deepseek](), [qwen]() Tags: [ai-inference](), [artificial-intelligence](), [data](), [deepseek](), [developer-tools](), [fine-tuning](), [hardware](), [inference](), [large-language-model](), [production](), [qwen](), [tokens](), [vllm]() ### AI overview This article explains how FastMTP-style fine-tuning improves vLLM speculative decoding. It describes using native multi-token prediction heads as speculators, adapting a single head for recursive multi-step drafting, extracting weights from verifier checkpoints, and producing vLLM-ready checkpoints without training from scratch. ### Source excerpt Autoregressive decoding makes large language model (LLM) inference memory-bandwidth bound: every token needs 1 full forward pass over billions of parameters, so the hardware spends most of its time moving weights rather than computing. MTP is a training objective: models like the DeepSeek and Qwen families learn to predict several future tokens at each position, which improves their data efficiency and quality. The post Optimize vLLM speculative decoding with FastMTP heads appeared first on Red Hat Developer. ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## Hot Chips 2026: Applying High Bandwidth Flash (HBF) DevFeed: [Hot Chips 2026: Applying High Bandwidth Flash (HBF)]() Original publisher: [Read original article]() Author: Chester Lam Published: 2026-08-23T22:51:05Z Content type: article Language: en Sources: [Chips and Cheese]() Topics: [Machine Learning & Artificial Intelligence](), [Inference](), [vllm](), [moe](), [Cache]() Tags: [2026](), [dram](), [machine-learning](), [memory](), [moe](), [ssd](), [vllm]() ### AI overview This article examines how High Bandwidth Flash (HBF) could support machine learning workloads. HBF does not yet have products; the discussion uses simulations and projections to explore software strategies, including moving Mixture-of-Experts components or KV cache data between HBF and faster memory, with vLLM as an example. ### Source excerpt Machine learning workloads have an insatiable appetite for DRAM capacity. Flash memory is cheaper per gigabyte of capacity than DRAM. Could it offer a way out? ## EP223: Ollama vs vLLM vs SGLang DevFeed: [EP223: Ollama vs vLLM vs SGLang]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-08-22T15:31:34Z Content type: comparison Language: en Sources: [ByteByteGo]() Topics: [Ollama](), [sglang](), [vllm](), [Large Language Model](), [API](), [Cache](), [Agent Skills]() Tags: [ai-agents](), [api](), [cache](), [models](), [ollama](), [sglang](), [vllm]() ### AI overview This comparison explains how Ollama, vLLM, and SGLang handle requests for open-weight models. Ollama uses a FIFO queue and pre-quantized GGUF models for local development and prototyping; vLLM uses continuous batching and PagedAttention for high-traffic serving; and SGLang uses prefix-aware scheduling and RadixAttention for agents, multi-turn chats, and structured outputs. ### Source excerpt To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. ## Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e DevFeed: [Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e]() Original publisher: [Read original article]() Author: Noble Ackerson Published: 2026-08-04T23:28:06Z Content type: tutorial Language: en Sources: [Google Developer Experts - Medium]() Topics: [Monitoring](), [vllm](), [Cloud](), [hugging face](), [Large Language Model](), [Claude](), [gemma](), [Security](), [Benchmark](), [Artificial Intelligence](), [incident]() Tags: [ai](), [ai-agent](), [artificial-intelligence](), [benchmark](), [claude](), [cybersecurity](), [evaluation](), [gemma](), [google-cloud-platform](), [hugging-face](), [incident](), [infrastructure](), [monitoring](), [responsible-ai](), [security](), [vllm]() ### AI overview This article describes batch-evaluating LLM agent trajectories for responsible-AI checks before incidents occur. It connects reported production-system breaches with Hugging Face's use of LLM-driven analysis over more than 17,000 attacker events, then presents a scheduled approach using Gemma through vLLM on Cloud TPU v5e. ### Source excerpt Batch-evaluating agent trajectories on Cloud TPU v5e (compliance-at-scale, part 2) Trajectory batch eval pipeline for rai-checklist-cli A week or so ago, Hugging Face disclosed that an autonomous agent had broken into its production infrastructure. Five days later, OpenAI confirmed the agent was theirs: a combination of its own models, running an internal cyber-capability eval with the production safety classifiers switched off. The models were being tested on a benchmark called ExploitGym. The fastest observable path to a solution ran through the answer key. They escaped the isolated environment through a package-registry proxy, chained stolen credentials with zero-day vulnerabilities, and pulled the test solutions out of Hugging Face's production database. Per Axios, the agent kept pursuing its assigned objective even after it had escaped the test environment. Nine days later, Anthropic said hold my beer, checked its own logs and found three more. It reviewed 141,006 runs and found three cases where Claude models had reached the open internet and breached real production systems, the earliest dating to April. Two of the three organizations learned about it when Anthropic notified them. One lab looked and found something. A second lab looked and found something. That is the whole story here, and it should be the uncomfortable part: none of this surfaced through production monitoring. It surfaced because somebody went back and read the trajectories. Nobody has published what score that run produced. It doesn't matter. The part of this story that matters for this series is what Hugging Face did next with their findings. To reconstruct the intrusion, Hugging Face's security team ran LLM-driven analysis agents over the full attacker action log: more than 17,000 recorded events. Reporting indicates they did that analysis with an open-weight model on their own infrastructure, partly so no hosted safety classifier sat between the responders and the attack data, and partly ## GenRec: Towards LLM-Native Recommendation at Netflix DevFeed: [GenRec: Towards LLM-Native Recommendation at Netflix]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-07-30T20:10:15Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [recommendation systems](), [Large Language Model](), [LLM Techniques](), [personalization](), [vllm]() Tags: [data](), [feature-engineering](), [genai](), [large-language-models](), [large-language-models-llms](), [llm](), [llms](), [netflix](), [personalization](), [ranking](), [recommendation-system](), [vllm]() ### AI overview Netflix presents GenRec, an LLM-backed recommendation ranker trained on Netflix-specific data and objectives. It converts user histories, item metadata, and context into text, uses a catalog-aware scoring head, aligns recommendations with long-term member value and business goals, and runs in prefill-only mode on Netflix's LLM serving stack. In a large-scale A/B test, GenRec improved short- and long-term online metrics while using fewer labeled examples and input signals than a mature production ranker. ### Source excerpt Authors: Ying Li, Arjun Rao, Shradha Sehgal Introduction Recommendations sit at the heart of the Netflix experience. Our current production models rely on thousands of hand-crafted features over users, items, and interactions, along with specialized architectures for sequence modeling, feature interactions, and multi-task objectives. This stack has evolved over many years to support diverse content types (movies, series, games, live, podcasts) and product surfaces, but its complexity makes it costly to onboard new use cases: adding a content type or surface can require significant feature engineering, architecture change, infrastructure work, and experimentation. At the same time, large language models (LLMs) are changing how we think about recommendation, as shown by recent work such as PLUM, GLIDE, and OneRec-Think. Their broad world knowledge and strong language understanding make it possible to represent user histories and item metadata directly as text, capture rich relationships in a shared semantic space, and steer recommendations via natural-language prompts. However, off-the-shelf LLMs are still far from production-ready recommenders: they often over-recommend globally popular content, hallucinate out-of-catalog items, ignore business constraints, and provide only limited personalization. To address this, we built GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec shows that an LLM-based ranker can match or exceed a mature production system while relying on far fewer labeled examples and input signals. Figure 1: GenRec pipeline. Raw logs of user history, item metadata, and context are transformed via context engineering into natural-language prompts and fed into the GenRec, which runs on vLLM in prefill-only mode and outputs scores for each catalog item, yielding a recommendation ranking. At a high level, GenRec: Verbalizes user histories, item metadata, and context as text ## How DigitalOcean Served Kimi K3 on Day Zero DevFeed: [How DigitalOcean Served Kimi K3 on Day Zero]() Original publisher: [Read original article]() Author: Shree Murthy Published: 2026-07-30T17:10:40Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Digital Ocean](), [Inference](), [GPU](), [vllm](), [Hardware](), [NVLink](), [OpenAI]() Tags: [benchmarks](), [digitalocean](), [distributed](), [engineering](), [gpu](), [inference](), [nvidia](), [nvlink](), [reasoning](), [streaming](), [vllm]() ### AI overview DigitalOcean describes how it served the Kimi K3 model on its Inference Engine from day zero, including GPU selection, distributed serving with llm-d, vLLM tuning, and verification against Moonshot AI's benchmarks. ### Source excerpt DigitalOcean launched Kimi K3 on day 0. It's already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a model this size running well on day zero took real work across several teams. Thanks to Moonshot AI, Inferact, RadixArk, NVIDIA, and AMD for the help getting there. Standing up a new model, integrating it into DigitalOcean's Inference Engine, and showcasing its unique attributes on day 0 takes three things: the right hardware, a tuned serving stack, and rigorous verification against Moonshot's own benchmarks. Here are the lessons we learned along the way: Hardware selection and implementation We selected NVIDIA HGX™ B300 and AMD Instinct™ MI350x GPUs to run K3 because these instances provide the memory capacity, FLOPs, and interconnect horsepower necessary for a model of K3's size and architecture. We built our distributed inference stack with llm-d because it includes native support for GPU type heterogeneity. This let us quickly onboard K3 to both AMD and NVIDIA platforms. Kimi K3 has roughly 2.78 trillion total parameters, 896 routed experts, and an attention stack that interleaves 69 Kimi Delta Attention (KDA) layers with 24 Gated Multi-head Latent Attention (MLA) layers. Kimi-K3 weights are ~1.56 TB in total, which requires about 195 GiB per GPU. Given such a large memory footprint for the weights alone, and a need to keep enough headroom for KV cache and activations, the practical unit of deployment is an 8x NVIDIA HGX B300 or AMD Instinct MI350X server. Both have 288GB of VRAM capacity, and after loading the weights, there is still some amount of practical memory left for the KV cache. Entire weights cannot be loaded on a single GPU. That's where the high-speed scaled-up NVIDIA's NVLink or AMD's Infinity Fabric is critical to ensure there is enough interconnect horsepower for bandwidth intensive, latency sensitive attention and expert parallel computations. Model ## Инференс LLM: от KV-кэша до продакшен-деплоя DevFeed: [Инференс LLM: от KV-кэша до продакшен-деплоя]() Original publisher: [Read original article]() Author: a\_ryzhov (hh.ru, Конференции Олега Бунина (Онтико)) Published: 2026-07-27T05:30:45Z Content type: tutorial Language: ru Sources: [HeadHunter RU]() Topics: [Large Language Model](), [Caching](), [on-prem](), [genai](), [MLOps](), [sglang](), [vllm]() Tags: [2026](), [compute](), [genai](), [http](), [inference](), [kv-cache](), [llm](), [mlops](), [on-prem](), [sglang](), [sram](), [vllm]() ### AI overview This Russian-language developer article explains how LLM inference behaves in on-premises production environments in 2026. It argues that GPU memory management is the main efficiency constraint, describes how KV caching shifts decoding from compute-bound to memory-bandwidth-bound work, and introduces vLLM and SGLang as ways to address the problem. ### Source excerpt Привет! Я Саша Рыжов, MLOps-инженер в hh.ru, уже три года занимаюсь развитием инфраструктуры для искусственного интеллекта. Компании, которые развивают GenAI, рано или поздно приходят к задачам по запуску LLM на собственном железе. В статье я расскажу, как обстоят дела с движками инференса в 2026 году и как запустить on-prem-прод и не изобрести при этом велосипед. Читать далее ## Synthetic Data Generation for Financial AI Research with NVIDIA NeMo DevFeed: [Synthetic Data Generation for Financial AI Research with NVIDIA NeMo]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-07-09T19:40:37Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Synthetic Data Generation](), [synthetic-data](), [Large Language Model](), [Fine-tuning](), [Natural language processing](), [Nemotron](), [vllm](), [datasets]() Tags: [agentic-ai-generative-ai](), [ai](), [ai-agent](), [ai-ready-data](), [cloud-services](), [compute](), [data](), [data-science](), [featured](), [financial-services](), [fine-tuning](), [generation](), [hardware](), [inference](), [llms](), [models](), [natural-language-processing](), [nemo](), [nemotron](), [nvidia](), [research](), [structured-generation](), [synthetic](), [synthetic-data](), [synthetic-data-generation](), [vllm](), [workflow]() ### AI overview This developer article presents an iterative pipeline for generating a diverse synthetic dataset of more than 500,000 financial news headlines. It combines NeMo Data Designer for structured generation, NeMo Curator for semantic deduplication, Nemotron models for synthesis, and a farthest-from-centroid few-shot strategy to reduce repetition and correct category imbalance. ### Source excerpt Fine-tuning LLMs for financial natural language processing (NLP) is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings... ## Native-speed vLLM transformers modeling backend DevFeed: [Native-speed vLLM transformers modeling backend]() Original publisher: [Read original article]() Author: Harry Mellor; Lysandre Published: 2026-07-08T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vllm](), [Inference Performance](), [Inference](), [LLMs](), [Machine learning](), [Open Source]() Tags: [architecture](), [batching](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [mixture-of-experts](), [mlx](), [open-source](), [optimization](), [sglang](), [transformers](), [vllm]() ### AI overview This article describes the transformers modeling backend for vLLM and its role in serving Hugging Face models without requiring model authors to port their implementations. It explains how transformers supplies modeling code while vLLM contributes optimized inference techniques, and outlines comparisons with vLLM's native implementations across several Qwen3 model configurations. It also discusses inference-performance factors including GPU parallelization, compilation, fused kernels, and custom attention kernels. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Run a vLLM Server on HF Jobs in One Command DevFeed: [Run a vLLM Server on HF Jobs in One Command]() Original publisher: [Read original article]() Author: Quentin Gallouédec Published: 2026-06-26T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [jobs](), [vllm](), [API](), [inference-endpoints](), [model-deployment](), [cURL](), [GPU](), [Python]() Tags: [api](), [gpu](), [guide](), [inference](), [inference-endpoints](), [infrastructure](), [jobs](), [json](), [model](), [models](), [python](), [vllm]() ### AI overview A practical guide to launching a vLLM model server on Hugging Face Jobs with a single command, querying it through the OpenAI-compatible API, authenticating requests with an HF token, managing costs, and scaling to larger multi-GPU models. It also contrasts ephemeral Jobs with managed Inference Endpoints. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Ubuntu Server on the NVIDIA DGX Spark (Without the Desktop) DevFeed: [Ubuntu Server on the NVIDIA DGX Spark (Without the Desktop)]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-06-22T13:00:00Z Content type: tutorial Language: en Sources: [Techno Tim]() Topics: [DGX Spark](), [Ubuntu](), [NVIDIA DGX](), [GPU](), [NVLink](), [CUDA](), [Docker](), [Ansible](), [vllm](), [Blackwell](), [networking](), [GitHub]() Tags: [ai](), [ansible](), [blackwell](), [cuda](), [dell](), [dgx-spark](), [docker](), [github](), [gpu](), [homelab](), [networking](), [nvidia-dgx](), [nvlink](), [ubuntu](), [vllm]() ### AI overview A guide to replacing DGX OS with a minimized Ubuntu 24.04 Server installation on GB10 systems such as the NVIDIA DGX Spark and ASUS Ascent GX10. It explains the memory and power benefits of removing GNOME while retaining the NVIDIA drivers, CUDA, Docker, and NVIDIA Container Toolkit, and covers ConnectX-7 networking, dual-node setup, and Ansible automation. ### Source excerpt When you buy an NVIDIA DGX Spark or an ASUS Ascent GX10, it ships with DGX OS. DGX OS is NVIDIA's managed Ubuntu image, and it is fine - if you want a full GNOME desktop on an AI box. I did not want that. The GB10 has 128 GB of unified memory shared between the CPU and GPU over NVLink-C2C. Every gigabyte the OS and desktop environment consume is a gigabyte not available to your model. On the ... ## Prefix-Aware Routing and Caching Reduce Redundant LLM Inference Costs DevFeed: [Prefix-Aware Routing and Caching Reduce Redundant LLM Inference Costs]() Original publisher: [Read original article]() Author: Simon Mo, CEO of Inferact Published: 2026-06-01T19:30:00Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Inference](), [Caching](), [Large Language Model](), [vllm](), [networking](), [Digital Ocean](), [Serverless]() Tags: [ai-ml](), [caching](), [digitalocean](), [engineering](), [gateway](), [gpu](), [gpu-optimization](), [inference](), [llms](), [serverless](), [vllm]() ### AI overview This article explains how repeated prompt prefixes create avoidable compute costs during LLM inference. It describes vLLM prefix caching and prefix-aware routing in DigitalOcean's inference gateway as ways to reduce redundant prefill work and GPU compute waste, including a claimed reduction of up to 4x on the same hardware. ### Source excerpt Introduction Inference demand is growing fast, and it's only accelerating. By 2030, inference is expected to account for the majority of AI compute globally. But scaling inference isn't just a hardware problem. Most teams discover too late that a significant portion of their compute spend is avoidable, primarily because their systems are silently repeating work they have already done, recomputing the same prompt prefixes and system instructions over and over again. We've seen this from two vantage points. From the infrastructure layer, the cost curve becomes visible at scale with clusters that look busy but aren't efficiently utilized. From the engine layer, the picture is just as clear. Without the right caching and scheduling primitives, even a well-optimized model wastes cycles on redundant computation. The root cause is the same regardless of where you're standing. The system lacks the memory and coordination to recognize when it's already done the hard part. Fixing this requires work at every layer of the stack. DigitalOcean has invested in GPU optimization across multiple fronts, from vLLM parallelism and quantization tuning to hardware-level kernel work. But one technique has had an outsized impact on cost efficiency at scale: prefix-aware routing and caching. In this post, we walk through how vLLM enables advanced prefix caching, how DigitalOcean's inference gateway uses prefix awareness to make smarter routing decisions, and how we plan to make this available to everyone on Serverless Inference in the coming weeks. The Cost Cliff and the Hidden Culprit Inference now accounts for roughly 70% of total AI compute costs. For most teams, a significant share of that is avoidable. It's not due to hardware limits. Instead, it's because the system keeps recomputing work it has already done, also known as redundant prefill. Every LLM inference request has two distinct computational phases. The first phase is prefill, where the model processes the entire input sequenc ## Running the Latest vLLM on the NVIDIA DGX Spark DevFeed: [Running the Latest vLLM on the NVIDIA DGX Spark]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-21T13:00:00Z Content type: tutorial Language: en Sources: [Techno Tim]() Topics: [vllm](), [DGX Spark](), [Docker](), [Dockerfile](), [GitHub Actions](), [NCCL](), [CUDA](), [PyTorch](), [GitHub](), [NGC](), [GPU](), [Ubuntu]() Tags: [ai](), [cuda](), [dgx-spark](), [docker](), [docker-image](), [github](), [github-actions](), [gpu](), [homelab](), [nccl](), [ngc](), [nvidia-dgx](), [pytorch](), [self-hosted](), [ubuntu](), [vllm]() ### AI overview This tutorial presents a reproducible Docker image pipeline for running current vLLM releases on the NVIDIA DGX Spark's GB10 ARM64 platform. It explains why NVIDIA's NGC images can lag upstream and describes a GitHub Actions build on a self-hosted Spark runner with pinned inputs and generated lockfiles. ### Source excerpt When I built my local AI cluster on a pair of ASUS Ascent GX10s, the hard part was not serving a model. The hard part was getting a working vLLM image with current components. NVIDIA's official image was already over a month behind by the time I needed it, and waiting on their release schedule was not an option. If you saw that post, you know the GX10 is an ARM64 machine built around NVIDIA's ... ## I Built a 256GB Local AI Cluster on My Desk DevFeed: [I Built a 256GB Local AI Cluster on My Desk]() Original publisher: [Read original article]() Author: Techno Tim Published: 2026-05-18T13:00:00Z Content type: article Language: en Sources: [Techno Tim]() Topics: [Local AI](), [Artificial Intelligence](), [agentic-coding](), [Hardware](), [Homelab](), [coding](), [model-serving](), [vllm](), [networking](), [Docker](), [NCCL](), [Monitoring]() Tags: [agentic-coding](), [ai](), [docker](), [github](), [hardware](), [homelab](), [local-ai](), [model-serving](), [monitoring](), [nccl](), [networking](), [self-hosted](), [self-hosted-ai](), [server](), [vllm]() ### AI overview The article tests a two-node local AI cluster built from ASUS Ascent GX10 systems, using one machine for practical agentic coding and two machines to run a larger local model. It evaluates a real coding workflow involving model serving, storage, networking, memory, Docker, NCCL, monitoring, power draw, and the application being built, concluding that local AI is capable and cloud-independent but still operationally complex. ### Source excerpt I have been covering local and self-hosted AI for a few years now - from running models privately at home to what is still running in my homelab today. But to run the larger, more capable models, you need something more specialized than a general-purpose home server. I wanted to know how good local AI has actually gotten, so I built a mini AI cluster on my desk and used it for a real coding wo... ## vLLM V0 to V1: Correctness Before Corrections in RL DevFeed: [vLLM V0 to V1: Correctness Before Corrections in RL]() Original publisher: [Read original article]() Author: Rafael Pardinas; Ehsan Kamalloo Published: 2026-05-06T19:06:55Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vllm](), [Back end](), [Inference](), [benchmarking](), [Optimization]() Tags: [backend](), [comparison](), [inference](), [optimization](), [rl](), [vllm]() ### AI overview This article explains how a vLLM V1 migration was brought to parity with a vLLM V0 reference for reinforcement-learning training. The authors fixed processed rollout logprobs, V1 runtime defaults, the inflight weight-update path, and final-projection precision before changing the RL objective. ### Source excerpt TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 used for the final projection. We fixed the backend behavior before changing the RL objective. The reference run used vLLM ; the V1 runs used vLLM . Figure 1 shows the final result. The red run is the initial V1 attempt, and the green run is the final V1 run after the fixes described below. ## Load Balancing and Scaling LLM Serving DevFeed: [Load Balancing and Scaling LLM Serving]() Original publisher: [Read original article]() Author: Mohammad Ashar Khan Published: 2026-04-15T19:03:31Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Load Balancing](), [Caching](), [Large Language Model](), [Routing (disambiguation)](), [Latency](), [sglang](), [vllm](), [Concurrency](), [TensorRT]() Tags: [caching](), [concurrency](), [efficiency](), [engineering](), [http](), [inference](), [latency](), [llm](), [load-balancing](), [performance](), [round-robin](), [sglang](), [tensorrt](), [vllm]() ### AI overview This article explains why scaling LLM serving requires load-balancing strategies that account for prompt-prefix caching. It describes how naive round-robin routing reduces cache-hit probability as the number of replicas grows, and introduces cache-aware routing approaches intended to preserve cache efficiency, control costs, and improve performance. It also discusses inference engines including vLLM, SGLang, and TensorRT, along with prefill and decode processing. ### Source excerpt Load balancing for LLMs is fundamentally different from load balancing for traditional services like web servers, APIs, or databases. Prompt caching is the reason. Prompt caching typically cuts input token costs by 50-90% and can reduce Time to First Token (TTFT) latency by up to 80%, but those gains assume your request lands on the replica that already has the relevant prefix cached. Under naive round-robin load balancing across N replicas, that probability is 1/N. The cache hit rate that made caching so attractive at one replica degrades almost linearly as your fleet grows. Solving this requires rethinking how requests are routed at the infrastructure level. This article covers the load balancing strategies and specialized routers that preserve cache efficiency at scale, starting with why standard approaches fall short and progressing to precise, cache-aware routing techniques. Inferencing engines To achieve large-scale inferencing, we use inference engines. These engines simplify the complexities of serving LLMs and offer improved resource utilization on the underlying GPUs. They also enable higher concurrency and allow for customization to suit diverse inference workloads, such as real-time chat completions and long-form document summarization. Noteworthy engine options include vLLM, SGLang, and TensorRT. The inferencing process is largely consistent across different engines. Sending an HTTP request to an engine initiates a standard sequence of steps. Prefill Phase: The input prompt is first converted into token IDs using the model's tokenizer. Requests are grouped into batches for efficient concurrent processing by the engine. During this initial processing, special Key (K) and Value (V) tensors are computed. This phase concludes after the first forward pass, resulting in the generation of the first output token. Decode Phase: This phase involves an auto-regressive loop, continuing until an end-of-sequence token is generated or the maximum sequence length is re ## Sheaf: vLLM for Non-Text Foundation Models DevFeed: [Sheaf: vLLM for Non-Text Foundation Models]() Original publisher: [Read original article]() Published: 2026-04-14T00:00:00Z Content type: article Language: en Sources: [Alex Korbonits]() Topics: [vllm](), [Inference](), [foundation-models](), [Large Language Model](), [Machine learning](), [model-deployment](), [Python](), [Streaming](), [Docker](), [lora](), [SIEM, Security, Observability]() Tags: [docker](), [foundation-models](), [gpu](), [inference](), [llms](), [lora](), [observability](), [python](), [streaming](), [vllm]() ### AI overview This article introduces Sheaf, a serving framework for non-text foundation models. It argues that vLLM's serving optimizations benefit from the shared compute pattern of autoregressive text LLMs, while time-series, tabular, molecular, biological, and diffusion models require different batching, memory-management, and inference approaches. The article describes Sheaf's proposed capabilities, including typed contracts, model-aware batching, streaming, caching, observability, offline batch inference, asynchronous workers, LoRA adapter multiplexing, a typed Python client, Docker and KubeRay deployment support, and 27 PyPI backends. ### Source excerpt vLLM solved inference for text LLMs. The same gap exists for every other class of foundation model -- time series, tabular, molecular, diffusion, and more. Sheaf fills it: typed contracts, model-type-aware batching, streaming, caching, observability, offline batch inference, an async-job worker, LoRA adapter multiplexing, a typed Python client, a Docker base image with KubeRay deployment, and 27 backends on PyPI. ## Advanced Prompt Caching at Scale DevFeed: [Advanced Prompt Caching at Scale]() Original publisher: [Read original article]() Author: Andrew Dugan Published: 2026-04-07T19:11:40Z Content type: tutorial Language: en Sources: [DigitalOcean]() Topics: [Caching](), [Inference Performance](), [Large Language Model](), [Inference](), [Latency](), [Load Balancing](), [round robin](), [sglang](), [TensorRT-LLM](), [vllm](), [model architecture]() Tags: [ai-ml](), [architecture](), [caching](), [decoding](), [efficiency](), [engineering](), [inference](), [latency](), [llm](), [load-balancing](), [prompt](), [round-robin](), [sglang](), [tensorrt-llm](), [token](), [tokens](), [vllm]() ### AI overview This tutorial explains how prompt caching works across multiple LLM replicas. It describes how round-robin load balancing reduces cache-hit rates and presents session affinity, tiered routing, and prefix-aware load balancing as architectural strategies for preserving KV-cache reuse while reducing latency and inference costs. ### Source excerpt Introduction Prompt caching is the process of reusing already computed KV states across inference requests in order to save money and reduce latency. Within a single replica, modern inference engines like vLLM, SGLang, and TensorRT-LLM handle it automatically. Incoming prompts are matched against cached prefixes and recomputed only where necessary, without requiring user configurations The problem nobody talks about is what happens when you scale to many replicas. Under round-robin load balancing, a request with an identical prefix has only a 1/N chance of hitting the replica where that prefix is already cached. The cache hit rate that made prompt caching so attractive at one replica degrades almost linearly as your fleet grows, unless you architect around it deliberately. Done right, prompt caching at scale offers 50-90% discounts on cached input tokens and can reduce time-to-first-token (TTFT) latency by up to 80%. This article covers the architectural strategies that make that possible. The Single-Replica Ceiling Refer to our previous prompt caching article for a detailed explanation of how KV caching works under the hood. Every transformer-based LLM uses KV caching to store key and value vectors from the attention layers in GPU VRAM during decoding. This intra-request caching is baked into the model architecture to increase throughput and maximize efficiency. Within a single replica, modern open-source engines like vLLM, SGLang (via RadixAttention), and TensorRT-LLM support automatic prefix caching out of the box, matching incoming prompts against previously cached prefixes to maximize KV reuse without any user configuration. Reusing KV states across requests from many users and replicas is where inference frameworks differ significantly. In the simplest architecture, the cache lives on individual replicas in VRAM. It is not shared across model instances at all. When a user makes an inference request, the prompt from their request is cached on a single replica. ## NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency DevFeed: [NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency]() Original publisher: [Read original article]() Author: Waverly Swinton Published: 2026-03-19T22:13:37Z Content type: release Language: en Sources: [DigitalOcean]() Topics: [Dynamo](), [Digital Ocean](), [Inference Performance](), [Artificial Intelligence](), [Generative AI](), [Inference](), [Cloud](), [GB200](), [sglang](), [TensorRT-LLM](), [vllm](), [Kubernetes]() Tags: [ai-ml](), [cloud](), [cost](), [deployment](), [digitalocean](), [dynamo](), [gb200](), [inference](), [inference-performance](), [kubernetes](), [latency](), [llm](), [load-balancing](), [nvidia](), [product-updates](), [routing](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview NVIDIA Dynamo 1.0 is now available to DigitalOcean customers as an inference service framework for large-scale generative AI workloads. The article describes claimed performance improvements on NVIDIA GB200 NVL systems, cost-efficiency benefits, deployment options, and features including GPU-aware routing and disaggregated serving. ### Source excerpt NVIDIA Dynamo 1.0, which was released on Monday at NVIDIA GTC, is now available to DigitalOcean customers to help drive performance enhancements and cost efficiency. NVIDIA Dynamo 1.0 offers a 7x inference performance increase on NVIDIA GB200 NVL systems, and by pairing it with DigitalOcean's Agentic Inference Cloud, customers can achieve higher performance at lower costs while benefiting from seamless deployment. Working together, DigitalOcean's optimizations with NVIDIA have already achieved a 67% cost savings for customers like Workato, and this new generation of Dynamo can unlock even greater gains for businesses who run production-grade agentic workflows. DigitalOcean customers can get access to NVIDIA Dynamo 1.0 as a container image that can be run on a Droplet or can deploy directly on DigitalOcean Kubernetes with an inference runtime (vLLM, SGlang, TensorRT). What is NVIDIA Dynamo 1.0? NVIDIA Dynamo is a cutting-edge, high-performance inference service framework specifically designed to accelerate and optimize large-scale generative AI and inference models. Dynamo is an orchestration layer that sits above engines like vLLM, SGLang, and NVIDIA TensorRT-LLM. Think of it as the distributed traffic controller for your GPU fleet, seamlessly orchestrating GPU and memory resources across a cluster and reducing bottleneck by intelligently routing requests Key technical breakthroughs offered by Dynamo 1.0 include: 7x Performance Boost: When paired with NVIDIA Blackwell Ultra GPUs, Dynamo can increase inference performance by up to 7x, significantly lowering your cost per token. KV-Aware Routing: Instead of simple round-robin load balancing, Dynamo routes requests to the specific GPUs that already have the relevant "memory" from previous turns of a conversation. Disaggregated Serving: Dynamo splits the "prefill" (reading the prompt) and "decode" (generating the answer) phases across different GPUs to maximize utilization and reduce latency. Memory Offloading: The KV B [Next page]()