# sglang High-performance serving framework for large language models and multimodal models. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation DevFeed: [How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation]() Original publisher: [Read original article]() Author: Claudio Masolo Published: 2026-09-11T10:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [recommendation systems](), [Large Language Model](), [Language models](), [sglang](), [Latency]() Tags: [agentic-ai-architecture](), [ai](), [ai-ml-data-engineering](), [artificial-intelligence](), [large-language-models](), [latency](), [liger](), [linkedin](), [linkedin-ai-multi-teacher](), [llms](), [ml-data-engineering](), [news](), [search](), [sglang](), [training]() ### AI overview LinkedIn describes a multi-teacher distillation pipeline for AI-powered job search that trains a 0.6B-parameter ranking model. The article focuses on SGLang-based teacher serving, online and offline distillation, and training optimizations reported to produce roughly an eightfold speedup. ### Source excerpt LinkedIn has published details of the training infrastructure behind its AI-powered job search, describing a multi-teacher distillation pipeline that compresses knowledge from large teacher models into a compact 0.6B-parameter ranking model. By Claudio Masolo ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## EP223: Ollama vs vLLM vs SGLang DevFeed: [EP223: Ollama vs vLLM vs SGLang]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-08-22T15:31:34Z Content type: comparison Language: en Sources: [ByteByteGo]() Topics: [Ollama](), [sglang](), [vllm](), [Large Language Model](), [API](), [Cache](), [Agent Skills]() Tags: [ai-agents](), [api](), [cache](), [models](), [ollama](), [sglang](), [vllm]() ### AI overview This comparison explains how Ollama, vLLM, and SGLang handle requests for open-weight models. Ollama uses a FIFO queue and pre-quantized GGUF models for local development and prototyping; vLLM uses continuous batching and PagedAttention for high-traffic serving; and SGLang uses prefix-aware scheduling and RadixAttention for agents, multi-turn chats, and structured outputs. ### Source excerpt To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. ## Up to 3.2x Faster Inference with LFM2.5-DSpark DevFeed: [Up to 3.2x Faster Inference with LFM2.5-DSpark]() Original publisher: [Read original article]() Author: Xx; Leonie Monigatti; Fernando Fernandes Neto; Tarek Dakhran; Nathan Ranchin Published: 2026-08-20T16:52:57Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Inference](), [Large Language Model](), [Latency](), [Streaming](), [sglang](), [GPU](), [Code]() Tags: [agentic](), [benchmark](), [code](), [gpu](), [inference](), [latency](), [llama](), [llama-cpp](), [llm](), [memory](), [on-device](), [streaming]() ### AI overview The article presents LFM2.5-DSpark, a speculative decoding approach that accelerates LLM inference by using a lightweight draft model and verifier. It reports up to 3.18x GPU throughput improvement and up to 2.87x on-device improvement, along with reduced function-calling latency and support for llama.cpp and SGLang. ### Source excerpt - Faster inference: up to 3.18 throughput improvement on a GPU and up to 2.87x on-device. - Toward on-device agentic inference: cuts function-calling latency by 57% on average for LFM2.5-2.6B - Day-one support for llama.cpp and SGLang: LFM-compatible DSpark integration is open-sourced upstream The decode phase in LLM inference is traditionally memory-bound. Most latency comes from streaming weights from DRAM into SRAM, not from intense computation. ## Инференс LLM: от KV-кэша до продакшен-деплоя DevFeed: [Инференс LLM: от KV-кэша до продакшен-деплоя]() Original publisher: [Read original article]() Author: a\_ryzhov (hh.ru, Конференции Олега Бунина (Онтико)) Published: 2026-07-27T05:30:45Z Content type: tutorial Language: ru Sources: [HeadHunter RU]() Topics: [Large Language Model](), [Caching](), [on-prem](), [genai](), [MLOps](), [sglang](), [vllm]() Tags: [2026](), [compute](), [genai](), [http](), [inference](), [kv-cache](), [llm](), [mlops](), [on-prem](), [sglang](), [sram](), [vllm]() ### AI overview This Russian-language developer article explains how LLM inference behaves in on-premises production environments in 2026. It argues that GPU memory management is the main efficiency constraint, describes how KV caching shifts decoding from compute-bound to memory-bandwidth-bound work, and introduces vLLM and SGLang as ways to address the problem. ### Source excerpt Привет! Я Саша Рыжов, MLOps-инженер в hh.ru, уже три года занимаюсь развитием инфраструктуры для искусственного интеллекта. Компании, которые развивают GenAI, рано или поздно приходят к задачам по запуску LLM на собственном железе. В статье я расскажу, как обстоят дела с движками инференса в 2026 году и как запустить on-prem-прод и не изобрести при этом велосипед. Читать далее ## Load Balancing and Scaling LLM Serving DevFeed: [Load Balancing and Scaling LLM Serving]() Original publisher: [Read original article]() Author: Mohammad Ashar Khan Published: 2026-04-15T19:03:31Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Load Balancing](), [Caching](), [Large Language Model](), [Routing (disambiguation)](), [Latency](), [sglang](), [vllm](), [Concurrency](), [TensorRT]() Tags: [caching](), [concurrency](), [efficiency](), [engineering](), [http](), [inference](), [latency](), [llm](), [load-balancing](), [performance](), [round-robin](), [sglang](), [tensorrt](), [vllm]() ### AI overview This article explains why scaling LLM serving requires load-balancing strategies that account for prompt-prefix caching. It describes how naive round-robin routing reduces cache-hit probability as the number of replicas grows, and introduces cache-aware routing approaches intended to preserve cache efficiency, control costs, and improve performance. It also discusses inference engines including vLLM, SGLang, and TensorRT, along with prefill and decode processing. ### Source excerpt Load balancing for LLMs is fundamentally different from load balancing for traditional services like web servers, APIs, or databases. Prompt caching is the reason. Prompt caching typically cuts input token costs by 50-90% and can reduce Time to First Token (TTFT) latency by up to 80%, but those gains assume your request lands on the replica that already has the relevant prefix cached. Under naive round-robin load balancing across N replicas, that probability is 1/N. The cache hit rate that made caching so attractive at one replica degrades almost linearly as your fleet grows. Solving this requires rethinking how requests are routed at the infrastructure level. This article covers the load balancing strategies and specialized routers that preserve cache efficiency at scale, starting with why standard approaches fall short and progressing to precise, cache-aware routing techniques. Inferencing engines To achieve large-scale inferencing, we use inference engines. These engines simplify the complexities of serving LLMs and offer improved resource utilization on the underlying GPUs. They also enable higher concurrency and allow for customization to suit diverse inference workloads, such as real-time chat completions and long-form document summarization. Noteworthy engine options include vLLM, SGLang, and TensorRT. The inferencing process is largely consistent across different engines. Sending an HTTP request to an engine initiates a standard sequence of steps. Prefill Phase: The input prompt is first converted into token IDs using the model's tokenizer. Requests are grouped into batches for efficient concurrent processing by the engine. During this initial processing, special Key (K) and Value (V) tensors are computed. This phase concludes after the first forward pass, resulting in the generation of the first output token. Decode Phase: This phase involves an auto-regressive loop, continuing until an end-of-sequence token is generated or the maximum sequence length is re ## Advanced Prompt Caching at Scale DevFeed: [Advanced Prompt Caching at Scale]() Original publisher: [Read original article]() Author: Andrew Dugan Published: 2026-04-07T19:11:40Z Content type: tutorial Language: en Sources: [DigitalOcean]() Topics: [Caching](), [Inference Performance](), [Large Language Model](), [Inference](), [Latency](), [Load Balancing](), [round robin](), [sglang](), [TensorRT-LLM](), [vllm](), [model architecture]() Tags: [ai-ml](), [architecture](), [caching](), [decoding](), [efficiency](), [engineering](), [inference](), [latency](), [llm](), [load-balancing](), [prompt](), [round-robin](), [sglang](), [tensorrt-llm](), [token](), [tokens](), [vllm]() ### AI overview This tutorial explains how prompt caching works across multiple LLM replicas. It describes how round-robin load balancing reduces cache-hit rates and presents session affinity, tiered routing, and prefix-aware load balancing as architectural strategies for preserving KV-cache reuse while reducing latency and inference costs. ### Source excerpt Introduction Prompt caching is the process of reusing already computed KV states across inference requests in order to save money and reduce latency. Within a single replica, modern inference engines like vLLM, SGLang, and TensorRT-LLM handle it automatically. Incoming prompts are matched against cached prefixes and recomputed only where necessary, without requiring user configurations The problem nobody talks about is what happens when you scale to many replicas. Under round-robin load balancing, a request with an identical prefix has only a 1/N chance of hitting the replica where that prefix is already cached. The cache hit rate that made prompt caching so attractive at one replica degrades almost linearly as your fleet grows, unless you architect around it deliberately. Done right, prompt caching at scale offers 50-90% discounts on cached input tokens and can reduce time-to-first-token (TTFT) latency by up to 80%. This article covers the architectural strategies that make that possible. The Single-Replica Ceiling Refer to our previous prompt caching article for a detailed explanation of how KV caching works under the hood. Every transformer-based LLM uses KV caching to store key and value vectors from the attention layers in GPU VRAM during decoding. This intra-request caching is baked into the model architecture to increase throughput and maximize efficiency. Within a single replica, modern open-source engines like vLLM, SGLang (via RadixAttention), and TensorRT-LLM support automatic prefix caching out of the box, matching incoming prompts against previously cached prefixes to maximize KV reuse without any user configuration. Reusing KV states across requests from many users and replicas is where inference frameworks differ significantly. In the simplest architecture, the cache lives on individual replicas in VRAM. It is not shared across model instances at all. When a user makes an inference request, the prompt from their request is cached on a single replica. ## NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency DevFeed: [NVIDIA Dynamo 1.0 Is Available to DigitalOcean Customers for Inference Performance and Cost Efficiency]() Original publisher: [Read original article]() Author: Waverly Swinton Published: 2026-03-19T22:13:37Z Content type: release Language: en Sources: [DigitalOcean]() Topics: [Dynamo](), [Digital Ocean](), [Inference Performance](), [Artificial Intelligence](), [Generative AI](), [Inference](), [Cloud](), [GB200](), [sglang](), [TensorRT-LLM](), [vllm](), [Kubernetes]() Tags: [ai-ml](), [cloud](), [cost](), [deployment](), [digitalocean](), [dynamo](), [gb200](), [inference](), [inference-performance](), [kubernetes](), [latency](), [llm](), [load-balancing](), [nvidia](), [product-updates](), [routing](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview NVIDIA Dynamo 1.0 is now available to DigitalOcean customers as an inference service framework for large-scale generative AI workloads. The article describes claimed performance improvements on NVIDIA GB200 NVL systems, cost-efficiency benefits, deployment options, and features including GPU-aware routing and disaggregated serving. ### Source excerpt NVIDIA Dynamo 1.0, which was released on Monday at NVIDIA GTC, is now available to DigitalOcean customers to help drive performance enhancements and cost efficiency. NVIDIA Dynamo 1.0 offers a 7x inference performance increase on NVIDIA GB200 NVL systems, and by pairing it with DigitalOcean's Agentic Inference Cloud, customers can achieve higher performance at lower costs while benefiting from seamless deployment. Working together, DigitalOcean's optimizations with NVIDIA have already achieved a 67% cost savings for customers like Workato, and this new generation of Dynamo can unlock even greater gains for businesses who run production-grade agentic workflows. DigitalOcean customers can get access to NVIDIA Dynamo 1.0 as a container image that can be run on a Droplet or can deploy directly on DigitalOcean Kubernetes with an inference runtime (vLLM, SGlang, TensorRT). What is NVIDIA Dynamo 1.0? NVIDIA Dynamo is a cutting-edge, high-performance inference service framework specifically designed to accelerate and optimize large-scale generative AI and inference models. Dynamo is an orchestration layer that sits above engines like vLLM, SGLang, and NVIDIA TensorRT-LLM. Think of it as the distributed traffic controller for your GPU fleet, seamlessly orchestrating GPU and memory resources across a cluster and reducing bottleneck by intelligently routing requests Key technical breakthroughs offered by Dynamo 1.0 include: 7x Performance Boost: When paired with NVIDIA Blackwell Ultra GPUs, Dynamo can increase inference performance by up to 7x, significantly lowering your cost per token. KV-Aware Routing: Instead of simple round-robin load balancing, Dynamo routes requests to the specific GPUs that already have the relevant "memory" from previous turns of a conversation. Disaggregated Serving: Dynamo splits the "prefill" (reading the prompt) and "decode" (generating the answer) phases across different GPUs to maximize utilization and reduce latency. Memory Offloading: The KV B ## Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason! DevFeed: [Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!]() Original publisher: [Read original article]() Author: Rakshith; Tom Peres Published: 2025-09-11T20:04:44Z Content type: news Language: en Sources: [Hugging Face - Blog]() Topics: [Inference](), [Chain-of-thought](), [LLM evaluation / benchmarking](), [qwen](), [Fine-tuning](), [Reinforcement learning](), [sglang](), [tgi](), [vllm]() Tags: [announce](), [benchmark](), [benchmarking](), [benchmarks](), [chain-of-thought](), [fine-tuning](), [inference](), [models](), [open](), [qwen](), [reasoning](), [reinforcement-learning](), [release](), [rl](), [sglang](), [tgi](), [vllm]() ### AI overview WRITER announces three open Palmyra-mini models ranging from 1.5B to 1.7B parameters: a lightweight base model and two reasoning variants. The models target efficient inference across varied applications, with GGUF and MLX quantizations available. The article reports benchmark results, describes Chain of Thought training for the reasoning variants, and discusses inference-framework compatibility and reinforcement-learning trade-offs. ### Source excerpt The team at WRITER is thrilled to announce the release of three new open models in the Palmyra-mini family. These models are designed to be powerful, lightweight, and highly performant for their size (1.5B to 1.7B), making them ideal for a wide range of applications with efficient inference. - palmyra-mini: A powerful, lightweight non-thinking base model. - palmyra-mini-thinking-a: A specialized variant optimized for complex reasoning and logic. ## Accelerate a World of LLMs on Hugging Face with NVIDIA NIM DevFeed: [Accelerate a World of LLMs on Hugging Face with NVIDIA NIM]() Original publisher: [Read original article]() Author: Neal Vaidya Published: 2025-07-21T18:01:30Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [NVIDIA NIM](), [hugging face](), [Deployment](), [Inference](), [Docker Container](), [TensorRT-LLM](), [sglang](), [vllm]() Tags: [cuda](), [deployment](), [docker](), [hugging-face](), [inference](), [llms](), [nim](), [nvidia-nim](), [sglang](), [tensorrt-llm](), [vllm]() ### AI overview This tutorial explains how NVIDIA NIM can deploy a broad range of LLMs from Hugging Face using a single Docker container. It covers supported checkpoint formats, inference frameworks, environment prerequisites, authentication, caching, permissions, and local model deployment. ### Source excerpt NVIDIA AI customers and ecosystem partners leverage NVIDIA NIM inference microservices to streamline deployment of the latest AI models on NVIDIA accelerated infrastructure, including LLMs, multi-modal and domain-specific models from NVIDIA, Meta, Mistral AI, Google and hundreds more innovative model builders. ## The Transformers Library: standardizing model definitions DevFeed: [The Transformers Library: standardizing model definitions]() Original publisher: [Read original article]() Author: Lysandre; Arthur Zucker; Pedro Cuenca; Julien Chaumond Published: 2025-05-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Transformer](), [LLMs](), [interoperability](), [sglang](), [vllm](), [llama.cpp](), [tgi](), [MLX]() Tags: [announcement](), [community](), [interoperability](), [library](), [llama-cpp](), [llms](), [mlx](), [open-source](), [sglang](), [tgi](), [transformers](), [vllm](), [vlms]() ### AI overview The article presents Transformers as a standard library for defining and supporting machine learning model architectures. It describes its broad ecosystem integrations, including training frameworks and inference engines, and highlights interoperability with vLLM, SGLang, TGI, llama.cpp, and MLX. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Open R1: Update #2 DevFeed: [Open R1: Update #2]() Original publisher: [Read original article]() Author: Loubna Ben Allal; Lewis Tunstall; Anton Lozhkov; Elie Bakouch; Guilherme Penedo; Hynek Kydlicek; Gabriel Martín Blázquez Published: 2025-02-10T16:10:47Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [deepseek](), [synthetic-data](), [datasets](), [Chain-of-thought](), [Fine-tuning](), [math](), [llama](), [qwen](), [sglang](), [vllm](), [math-verify](), [Parser]() Tags: [chain-of-thought](), [datasets](), [deepseek](), [fine-tuning](), [llama](), [math](), [math-verify](), [qwen](), [sglang](), [synthetic-data](), [vllm]() ### AI overview Open R1 Update #2 presents OpenR1-Math-220k, a large-scale mathematical reasoning dataset created to help reconstruct parts of the DeepSeek R1 training pipeline and synthetic-data process. The article describes reasoning-trace generation, local inference with vLLM and SGLang, automated filtering with Math Verify, and the use of Llama3.3-70B-Instruct as a judge. It also discusses distillation and fine-tuning of Qwen and Llama models using reasoning traces. ### Source excerpt We are now two weeks into the Open R1 project which aims to reconstruct the missing pieces of DeepSeek R1--specifically, the training pipeline and synthetic data. In this post, we are happy to share the construction of OpenR1-Math-220k: our first large-scale dataset for mathematical reasoning! ## Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference DevFeed: [Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference]() Original publisher: [Read original article]() Author: Morgan Funtowicz; Hugo Larcher Published: 2025-01-16T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [tgi](), [model-deployment](), [Rust](), [vllm](), [TensorRT-LLM](), [Concurrency](), [Memory Safety](), [servers](), [llama.cpp](), [sglang](), [Python](), [Code]() Tags: [backend](), [backends](), [community](), [concurrency](), [http](), [inference](), [jetstream](), [llama-cpp](), [memory-safety](), [neuron](), [open-source](), [performance](), [production](), [python](), [rust](), [sglang](), [tensorrt-llm](), [tgi](), [vllm]() ### AI overview Hugging Face introduces TGI Backends, a unified frontend architecture for integrating inference solutions such as vLLM, SGLang, llama.cpp, and TensorRT-LLM. The approach lets users switch backends based on model, hardware, and performance requirements while supporting production deployment across diverse accelerators. The article also describes TGI's Rust and Python components, including Rust-based HTTP and scheduling layers designed for memory safety and concurrency. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.