# vllm Published articles for vllm. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Catching poor LLM performance and accuracy before deployment DevFeed: [Catching poor LLM performance and accuracy before deployment]() Original publisher: [Read original article]() Author: Christopher Miyai Published: 2026-09-17T13:16:47Z Content type: article Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [vllm](), [Inference Performance](), [LLM evaluation / benchmarking](), [benchmarking](), [CI/CD](), [model-deployment](), [Open Source](), [container]() Tags: [ai-inference](), [benchmarking](), [benchmarks](), [ci-cd](), [container-images](), [deployment](), [developer-tools](), [llm](), [open-source](), [performance](), [vllm]() ### AI overview This article describes a regression-testing system for Red Hat AI Inference that compares upstream and midstream vLLM builds. It benchmarks LLM performance and accuracy across hardware variants and uses CI/CD to detect regressions before release. ### Source excerpt vLLM is the leading open source inference and serving engine for LLMs. Averaging 64 commits a day with bi-weekly releases, the open source project goes through a significant amount of code changes rapidly. Red Hat AI Inference offers an integrated inference platform powered by vLLM, llm-d and vLLM's llm-compressor. The post Catching poor LLM performance and accuracy before deployment appeared first on Red Hat Developer. ## NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut DevFeed: [NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut]() Original publisher: [Read original article]() Author: Zhihan Jiang Published: 2026-09-16T15:00:48Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [benchmarking](), [Inference Performance](), [Dynamo](), [vllm](), [Nvidia](), [TensorRT-LLM]() Tags: [ai-infrastructure](), [benchmarks](), [dynamo](), [efficiency](), [hardware](), [inference](), [mlperf](), [networking](), [nvidia](), [nvidia-vera-rubin](), [nvl72](), [optimization](), [performance](), [software](), [tensorrt](), [tensorrt-llm](), [vera-rubin-nvl72](), [vllm]() ### AI overview NVIDIA reports MLPerf Inference v6.1 preview results for Vera Rubin NVL72 and GB300 NVL72 systems. Vera Rubin NVL72 delivered up to 3.7x higher throughput than GB300 NVL72 on Qwen3-VL and up to 2.5x higher throughput on DeepSeek-R1, while a four-rack GB300 NVL72 submission achieved 99% scaling efficiency. The results used vLLM, NVIDIA Dynamo, and TensorRT-LLM. ### Source excerpt System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. [...] ## Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU DevFeed: [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU]() Original publisher: [Read original article]() Author: Anthony Su; Injae Kwak Published: 2026-09-12T11:04:33.891311Z Content type: article Language: en Sources: [Google Developers Blog]() Topics: [Embeddings](), [vllm](), [Inference](), [long-context](), [autoscaling](), [Kubernetes](), [Cloud](), [Google](), [Artificial Intelligence](), [model-deployment](), [Open Source]() Tags: [autoscaling](), [embedding](), [github](), [gpu](), [inference](), [kubernetes](), [long-context](), [multimodal](), [open-source](), [retrieval](), [tpu](), [vllm]() ### AI overview This article explains how Google Cloud integrated native TPU support into vLLM to serve long-context, multimodal embedding models at production scale. It describes elastic TPU provisioning with Google Kubernetes Engine, autoscaling across accelerator types, and TPU-specific optimizations for long sequences and chunked prefill. The resulting setup is designed to preserve numerical parity with GPU reference baselines while supporting high-throughput semantic retrieval applications. ### Source excerpt Google Cloud has natively integrated TPU support into the vLLM serving engine, allowing developers to elastically scale high-demand embedding pipelines using Google Kubernetes Engine (GKE). To handle massive 15K+ token contexts for models like Qwen3-Embedding-8B, the engineering team implemented TPU-specific optimizations such as hardware-safe tensor alignment, JAX/XLA compilation pre-warming, and a hybrid StepPool architecture for chunked prefill management. These enhancements achieve near-perfect numerical parity with reference GPU baselines, and developers can immediately leverage the open-sourced setup recipes on the AI-Hypercomputer GitHub to build their own high-throughput semantic retrieval applications. ## The Architecture for Serving 100 Fine-Tuned Models on One GPU DevFeed: [The Architecture for Serving 100 Fine-Tuned Models on One GPU]() Original publisher: [Read original article]() Author: Avi Chawla Published: 2026-09-11T21:25:15Z Content type: tutorial Language: en Sources: [Daily Dose of Data Science]() Topics: [GPU](), [vllm](), [Fine-tuning](), [lora](), [Serverless](), [Deployment](), [Code]() Tags: [code](), [deployment](), [fine-tuning](), [gpu](), [lora](), [memory](), [models](), [production](), [serverless](), [vllm](), [workers]() ### AI overview This tutorial compares architectures for serving 100 fine-tuned 7B model variants on GPUs. It explains how separate merged models increase storage, GPU memory use, scaling pools, cold starts, and idle capacity, while a shared base model with LoRA adapters enables adapter reuse through vLLM. The article plans to test merged, unmerged startup-loaded, request-time adapter loading, and hosted-per-tenant deployments on Runpod Serverless. ### Source excerpt ...explained with code. ## Building Pinterest's VLM Serving Stack on NVIDIA Dynamo DevFeed: [Building Pinterest's VLM Serving Stack on NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-10T23:08:16Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [vlm](), [Inference](), [model-deployment](), [Orchestration](), [Latency](), [Caching](), [Open Source Models & Datasets]() Tags: [blackwell](), [cache](), [dynamo](), [embeddings](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [multimodal-ai](), [nvidia](), [nvidia-blackwell](), [open-source](), [orchestration](), [performance](), [pinterest](), [vllm](), [vlm](), [vlm-serving]() ### AI overview Pinterest describes its VLM serving stack built on NVIDIA Blackwell GPUs and NVIDIA Dynamo. The stack addresses multimodal inference demands such as image processing, variable prefill costs, KV-cache pressure, routing, and cache offloading. ### Source excerpt Lei Pan | Senior Software Engineer; Salina Wu | Senior Software Engineer; Cristian Lopez | Software Engineer I; Guangtong Bai | Staff Software Engineer; Soam Acharya | Principal Engineer; Saurabh Vishwas Joshi | Principal Engineer; Chia-Wei Chen | Staff Software Engineer; Ambud Sharma | Principal Engineer Why VLM Serving Matters at Pinterest Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more. This direction also reflects Pinterest's broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance. Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest's VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today's most demanding AI workloads -- including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity -- that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that g ## Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:58:09Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Large Language Model](), [Low-Latency Inference](), [Caching](), [Inference](), [Inference Performance]() Tags: [advanced-300](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [announcements](), [benchmarks](), [caching](), [inference](), [latency](), [llama](), [llm](), [routing](), [tensorrt-llm](), [vllm]() ### AI overview Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token. ### Source excerpt Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%. ## From token consumer to token provider: Building your org's AI API DevFeed: [From token consumer to token provider: Building your org's AI API]() Original publisher: [Read original article]() Author: Markell Rawls Published: 2026-09-10T13:01:50Z Content type: tutorial Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [Artificial Intelligence](), [API](), [API keys](), [AI Infrastructure](), [Security](), [NVIDIA DGX](), [vllm](), [Claude Code](), [hugging face](), [anthropic](), [OpenAI]() Tags: [ai](), [ai-inference](), [ai-infrastructure](), [api](), [api-keys](), [claude](), [claude-code](), [developer](), [gpu](), [hugging-face](), [openai](), [security](), [vllm]() ### AI overview The article describes how organizations can move from independently purchasing AI-provider access and distributing API keys toward centrally managing AI infrastructure and providing internal AI APIs. It highlights rising token costs, fragmented workloads, limited visibility, and security-audit difficulties. ### Source excerpt One of the biggest problems with AI right now is that it's expensive. If you've been anywhere near an enterprise IT budget in the last 2 years, then you already know that. Most companies that wanted to get AI into their workflows did the same thing: They signed up for a business account with an AI provider, handed out API keys, and started building. It made sense at the time because the models were good, the APIs were simple, and the alternative was standing up your own inference infrastructure, which nobody had bandwidth for. But then the bills started coming in. The post From token consumer to token provider: Building your org's AI API appeared first on Red Hat Developer. ## Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL DevFeed: [Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL]() Original publisher: [Read original article]() Author: Amine Dirhoussi; Quentin Gallouédec; Kashif Rasul; Sergio Paniego Published: 2026-09-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [lora](), [grpo](), [jobs](), [vllm](), [NCCL](), [async](), [hugging face]() Tags: [async](), [grpo](), [hub](), [hugging-face](), [jobs](), [llm](), [lora](), [nccl](), [open-source](), [rl](), [storage](), [trl](), [vllm]() ### AI overview This article describes asynchronous GRPO training with a LoRA adapter across separate Hugging Face Jobs. The adapter is synchronized to vLLM replicas through a shared Storage Bucket, while a proxy handles authentication, rollout routing, and adapter-load broadcasts. Five runs reduced the time for 500 steps from 3 hours 27 minutes to 53 minutes. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM DevFeed: [Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM]() Original publisher: [Read original article]() Author: Dmitry Soldatkin Published: 2026-09-09T22:26:29Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [Deployment](), [Inference](), [Machine learning](), [Language models](), [GPU](), [long-context](), [Mixture of Experts (MoE)]() Tags: [advanced-300](), [agentic](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [deployment](), [gpu](), [how-to](), [inference](), [long-context](), [moe](), [nvfp4](), [quantization](), [qwen](), [reasoning](), [technical-how-to](), [tool](), [vllm]() ### AI overview A deployment walkthrough for serving the open-weight Qwen3.8-2.4T-A95B language model on Amazon SageMaker HyperPod with vLLM and NVIDIA B300 GPUs. It covers provisioning, NVFP4 quantization, an OpenAI-compatible endpoint, reasoning, tool calling, and MTP speculative decoding. ### Source excerpt Learn how to deploy Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight model, on Amazon SageMaker HyperPod with vLLM. This walkthrough covers cluster provisioning, NVFP4 quantization, and an OpenAI-compatible endpoint with built-in reasoning, tool calling, and native MTP speculative decoding. ## Optimize vLLM speculative decoding with FastMTP heads DevFeed: [Optimize vLLM speculative decoding with FastMTP heads]() Original publisher: [Read original article]() Author: Rahul Tuli Published: 2026-09-08T14:20:16Z Content type: article Language: en Sources: [Red Hat](), [Red Hat Developer]() Topics: [vllm](), [Inference](), [Large Language Model](), [Fine-tuning](), [model-deployment](), [deepseek](), [qwen]() Tags: [ai-inference](), [artificial-intelligence](), [data](), [deepseek](), [developer-tools](), [fine-tuning](), [hardware](), [inference](), [large-language-model](), [production](), [qwen](), [tokens](), [vllm]() ### AI overview This article explains how FastMTP-style fine-tuning improves vLLM speculative decoding. It describes using native multi-token prediction heads as speculators, adapting a single head for recursive multi-step drafting, extracting weights from verifier checkpoints, and producing vLLM-ready checkpoints without training from scratch. ### Source excerpt Autoregressive decoding makes large language model (LLM) inference memory-bandwidth bound: every token needs 1 full forward pass over billions of parameters, so the hardware spends most of its time moving weights rather than computing. MTP is a training objective: models like the DeepSeek and Qwen families learn to predict several future tokens at each position, which improves their data efficiency and quality. The post Optimize vLLM speculative decoding with FastMTP heads appeared first on Red Hat Developer. ## Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson DevFeed: [Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson]() Original publisher: [Read original article]() Author: Elizabeth Goodman Published: 2026-09-04T16:21:04Z Content type: tutorial Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Jetson](), [model-deployment](), [Inference](), [Inference Performance](), [Optimization](), [Mixture of Experts (MoE)]() Tags: [agentic-ai-generative-ai](), [edge](), [edge-computing](), [how-to](), [inference](), [inference-performance](), [jetpack](), [jetson](), [jetson-orin](), [mixture-of-experts-moe](), [nvfp4](), [optimization](), [physical-ai](), [reasoning](), [thor](), [tutorial](), [vllm]() ### AI overview A tutorial on deploying and optimizing compact reasoning and agentic AI models on NVIDIA Jetson. It covers choosing models, improving inference with NVFP4 quantization and speculative decoding, serving example models with vLLM, and validating a configuration for a workload. ### Source excerpt Running reasoning and agentic AI at the edge has been harder than it needs to be. Until recently, models capable of multi-step reasoning were too large to run... ## Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026 DevFeed: [Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026]() Original publisher: [Read original article]() Author: Gerardo Delgado Published: 2026-09-03T16:00:59Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Local AI](), [Inference](), [Hardware](), [Inference Performance](), [Mixture of Experts (MoE)]() Tags: [agentic-ai](), [agents](), [ai](), [ai-inference](), [announcements](), [artificial-intelligence](), [dgx-spark](), [hardware](), [inference](), [llama-cpp](), [local-ai](), [mixture-of-experts-moe](), [nvidia](), [nvidia-rtx](), [open-source](), [rtx-ai-garage](), [rtx-spark](), [video](), [vllm]() ### AI overview NVIDIA announces local-AI updates at IFA 2026, including agent tooling, faster local inference, RTX Spark Windows PCs, and locally runnable models for agentic, coding, and video-generation workloads. ### Source excerpt Frontier intelligence is going local. At IFA 2026, NVIDIA, Microsoft and its partners are teaming up to provide faster inference and new tools that make agents easier to set up and run locally on NVIDIA hardware. New compact NVIDIA RTX Spark Windows PCs are also coming in October to give AI enthusiasts, developers and creators [...] ## Fast model loading for AI inference on Amazon EKS DevFeed: [Fast model loading for AI inference on Amazon EKS]() Original publisher: [Read original article]() Author: Sajjan Gundapuneedi Published: 2026-09-01T15:48:15Z Content type: article Language: en Sources: [Containers]() Topics: [Inference Performance](), [Amazon Elastic Kubernetes Service](), [Amazon S3](), [Language models]() Tags: [advanced-300](), [ai](), [ai-inference](), [amazon-eks](), [amazon-elastic-kubernetes-service](), [best-practices](), [cuda](), [gpu](), [inference](), [kubernetes](), [language-models](), [pytorch](), [s3](), [sglang](), [startup](), [vllm]() ### AI overview The article analyzes cold-start delays for AI inference pods on Amazon EKS. It finds that startup bottlenecks vary by model size: torch.compile dominates for smaller models, while loading weights from S3 to GPU memory dominates for larger models. Configuration changes to Run:ai Model Streamer reduce model-loading time on repeat launches. ### Source excerpt When you scale AI inference on Amazon EKS, every new pod must load model weights into GPU memory before serving traffic. We investigated where cold-start time goes and found two configuration-only changes to Run:ai Model Streamer that cut model startup time by 80-93% on subsequent launches, with no code changes. ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## Granite 4.2 LLMs: How They're Built DevFeed: [Granite 4.2 LLMs: How They're Built]() Original publisher: [Read original article]() Author: Yousaf Shah; Swanand Kadhe; Riddhiman Moulick; Ashish Sunil Agrawal; Santosh Borse Published: 2026-08-25T15:14:14Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Language models](), [AI Chat](), [AI-assisted coding](), [releases]() Tags: [agentic](), [agents](), [apache](), [chain-of-thought](), [code](), [fine-tuning](), [grouped-query-attention](), [ibm](), [llms](), [models](), [post-training](), [reasoning](), [reinforcement-learning](), [sglang](), [tool](), [training](), [transformer-architecture](), [vllm]() ### AI overview Granite 4.2 is a family of 3B, 8B, and 30B dense decoder-only reasoning language models. The article covers their training pipeline, thinking modes, native tool calling, and agentic reinforcement learning for the 8B and 30B models. ### Source excerpt Authors: Granite Team, IBM TL;DR: Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: 3B, 8B, and 30B. These models are post-trained from Granite-4.1 base models. Granite-4.1 base models were pre-trained from scratch on roughly 15T tokens with a five-phase strategy that extends the context window to 512K tokens, supervised fine-tuned on chain-of-thought, reasoning, and agentic-trajectory data, then post-trained with a multi-stage reinforcement... ## Hot Chips 2026: Applying High Bandwidth Flash (HBF) DevFeed: [Hot Chips 2026: Applying High Bandwidth Flash (HBF)]() Original publisher: [Read original article]() Author: Chester Lam Published: 2026-08-23T22:51:05Z Content type: article Language: en Sources: [Chips and Cheese]() Topics: [Machine Learning & Artificial Intelligence](), [Inference](), [vllm](), [moe](), [Cache]() Tags: [2026](), [dram](), [machine-learning](), [memory](), [moe](), [ssd](), [vllm]() ### AI overview This article examines how High Bandwidth Flash (HBF) could support machine learning workloads. HBF does not yet have products; the discussion uses simulations and projections to explore software strategies, including moving Mixture-of-Experts components or KV cache data between HBF and faster memory, with vLLM as an example. ### Source excerpt Machine learning workloads have an insatiable appetite for DRAM capacity. Flash memory is cheaper per gigabyte of capacity than DRAM. Could it offer a way out? ## EP223: Ollama vs vLLM vs SGLang DevFeed: [EP223: Ollama vs vLLM vs SGLang]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-08-22T15:31:34Z Content type: comparison Language: en Sources: [ByteByteGo]() Topics: [Ollama](), [sglang](), [vllm](), [Large Language Model](), [API](), [Cache](), [Agent Skills]() Tags: [ai-agents](), [api](), [cache](), [models](), [ollama](), [sglang](), [vllm]() ### AI overview This comparison explains how Ollama, vLLM, and SGLang handle requests for open-weight models. Ollama uses a FIFO queue and pre-quantized GGUF models for local development and prototyping; vLLM uses continuous batching and PagedAttention for high-traffic serving; and SGLang uses prefix-aware scheduling and RadixAttention for agents, multi-turn chats, and structured outputs. ### Source excerpt To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. ## Meta is back with Muse Glimmer: local, agentic, multimodal, and open source DevFeed: [Meta is back with Muse Glimmer: local, agentic, multimodal, and open source]() Original publisher: [Read original article]() Author: Pedro Cuenca; merve; ben burtenshaw; Aritra Roy Gosthipaty Published: 2026-08-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [multimodal](), [Artificial Intelligence](), [hugging face](), [Benchmark](), [coding]() Tags: [agentic](), [artificial-intelligence](), [benchmark](), [hub](), [hugging-face](), [images](), [llama-cpp](), [llms](), [local](), [meta](), [model](), [multimodal](), [muse](), [open-source](), [videos](), [vllm](), [vlm](), [vlms]() ### AI overview Hugging Face presents Muse Glimmer, a local, agentic, multimodal, open-source 30B-parameter vision-language model developed with Meta. The article outlines its vision and language architecture, benchmark context, optional speculative decoding for faster generation, and support for both images and videos. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e DevFeed: [Batch-Evaluating LLM Agent Trajectories for Responsible AI Checks on Cloud TPU v5e]() Original publisher: [Read original article]() Author: Noble Ackerson Published: 2026-08-04T23:28:06Z Content type: tutorial Language: en Sources: [Google Developer Experts - Medium]() Topics: [Monitoring](), [vllm](), [Cloud](), [hugging face](), [Large Language Model](), [Claude](), [gemma](), [Security](), [Benchmark](), [Artificial Intelligence](), [incident]() Tags: [ai](), [ai-agent](), [artificial-intelligence](), [benchmark](), [claude](), [cybersecurity](), [evaluation](), [gemma](), [google-cloud-platform](), [hugging-face](), [incident](), [infrastructure](), [monitoring](), [responsible-ai](), [security](), [vllm]() ### AI overview This article describes batch-evaluating LLM agent trajectories for responsible-AI checks before incidents occur. It connects reported production-system breaches with Hugging Face's use of LLM-driven analysis over more than 17,000 attacker events, then presents a scheduled approach using Gemma through vLLM on Cloud TPU v5e. ### Source excerpt Batch-evaluating agent trajectories on Cloud TPU v5e (compliance-at-scale, part 2) Trajectory batch eval pipeline for rai-checklist-cli A week or so ago, Hugging Face disclosed that an autonomous agent had broken into its production infrastructure. Five days later, OpenAI confirmed the agent was theirs: a combination of its own models, running an internal cyber-capability eval with the production safety classifiers switched off. The models were being tested on a benchmark called ExploitGym. The fastest observable path to a solution ran through the answer key. They escaped the isolated environment through a package-registry proxy, chained stolen credentials with zero-day vulnerabilities, and pulled the test solutions out of Hugging Face's production database. Per Axios, the agent kept pursuing its assigned objective even after it had escaped the test environment. Nine days later, Anthropic said hold my beer, checked its own logs and found three more. It reviewed 141,006 runs and found three cases where Claude models had reached the open internet and breached real production systems, the earliest dating to April. Two of the three organizations learned about it when Anthropic notified them. One lab looked and found something. A second lab looked and found something. That is the whole story here, and it should be the uncomfortable part: none of this surfaced through production monitoring. It surfaced because somebody went back and read the trajectories. Nobody has published what score that run produced. It doesn't matter. The part of this story that matters for this series is what Hugging Face did next with their findings. To reconstruct the intrusion, Hugging Face's security team ran LLM-driven analysis agents over the full attacker action log: more than 17,000 recorded events. Reporting indicates they did that analysis with an open-weight model on their own infrastructure, partly so no hosted safety classifier sat between the responders and the attack data, and partly ## Adapting open source practices to an AI-first world: A retrospective on 2025 DevFeed: [Adapting open source practices to an AI-first world: A retrospective on 2025]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-08-03T18:30:00Z Content type: article Language: en Sources: [Google Open Source Blog]() Topics: [Open Source](), [Google](), [A2A protocol](), [interoperability](), [GitHub](), [linux foundation](), [Protocol (disambiguation)](), [LLVM](), [Rust](), [vllm](), [gRPC](), [Kubernetes]() Tags: [2025](), [a2a](), [a2a-protocol](), [agents](), [ai](), [ai-agents](), [github](), [google-open-source](), [interoperability](), [kubernetes](), [linux](), [linux-foundation](), [open-source](), [report-card](), [retrospective](), [rust](), [security](), [vllm]() ### AI overview Google reviews Alphabet's 2025 open source activity, including employee and external contributions, collaboration on the Agent2Agent protocol, and the maintenance and launch of public repositories and tools. ### Source excerpt by Sophia Vargas, Google Open Source Even as AI adoption accelerates and transforms the global technology landscape, open source remains foundational to how Alphabet builds, uses, and collaborates on products for billions of users. Our commitment to open source remains broad and consistent, including sharing our work year-over-year, and reflecting on what we've learned. In 2025: Roughly 10% of Alphabet's full-time workforce actively contributed to open source projects. This contribution ratio has remained steady over the past five years, scaling to match our growth. These open source contributions are not just solely focused on Google. Our top projects by unique contributors at Alphabet include community-led projects such as LLVM, vLLM, Envoy, and Rust, as well as Google-initiated projects like Kubernetes, Apache Beam, and gRPC. In addition, Alphabet projects received commits from more than 20,000 non-Alphabet affiliated user accounts. Working together on emerging standards Open source communities continue to provide vital collaborative spaces to define emerging standards, ensuring the interoperability and extensibility for the next generation of technologies. In 2025, we worked with more than 50 partners on the Agent2Agent (A2A) protocol to enable AI agents to communicate with each other, securely exchange information, and coordinate actions on top of various enterprise platforms and applications. Within weeks of our initial announcement, Google donated the A2A project to the Linux Foundation as part of our long-standing commitment to develop "open, collaborative ecosystem - offering greater autonomy and multiplying productivity." Launching tools with transparency Open source licenses provide a framework for anyone to explore, test, fork and expand on our technologies. Over the last 15 years, Google has created more than 15,000 public repositories on GitHub. Today, Google continues to maintain more than 5,000 public repositories on GitHub, and more than 1,500 publi ## GenRec: Towards LLM-Native Recommendation at Netflix DevFeed: [GenRec: Towards LLM-Native Recommendation at Netflix]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-07-30T20:10:15Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [recommendation systems](), [Large Language Model](), [LLM Techniques](), [personalization](), [vllm]() Tags: [data](), [feature-engineering](), [genai](), [large-language-models](), [large-language-models-llms](), [llm](), [llms](), [netflix](), [personalization](), [ranking](), [recommendation-system](), [vllm]() ### AI overview Netflix presents GenRec, an LLM-backed recommendation ranker trained on Netflix-specific data and objectives. It converts user histories, item metadata, and context into text, uses a catalog-aware scoring head, aligns recommendations with long-term member value and business goals, and runs in prefill-only mode on Netflix's LLM serving stack. In a large-scale A/B test, GenRec improved short- and long-term online metrics while using fewer labeled examples and input signals than a mature production ranker. ### Source excerpt Authors: Ying Li, Arjun Rao, Shradha Sehgal Introduction Recommendations sit at the heart of the Netflix experience. Our current production models rely on thousands of hand-crafted features over users, items, and interactions, along with specialized architectures for sequence modeling, feature interactions, and multi-task objectives. This stack has evolved over many years to support diverse content types (movies, series, games, live, podcasts) and product surfaces, but its complexity makes it costly to onboard new use cases: adding a content type or surface can require significant feature engineering, architecture change, infrastructure work, and experimentation. At the same time, large language models (LLMs) are changing how we think about recommendation, as shown by recent work such as PLUM, GLIDE, and OneRec-Think. Their broad world knowledge and strong language understanding make it possible to represent user histories and item metadata directly as text, capture rich relationships in a shared semantic space, and steer recommendations via natural-language prompts. However, off-the-shelf LLMs are still far from production-ready recommenders: they often over-recommend globally popular content, hallucinate out-of-catalog items, ignore business constraints, and provide only limited personalization. To address this, we built GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec shows that an LLM-based ranker can match or exceed a mature production system while relying on far fewer labeled examples and input signals. Figure 1: GenRec pipeline. Raw logs of user history, item metadata, and context are transformed via context engineering into natural-language prompts and fed into the GenRec, which runs on vLLM in prefill-only mode and outputs scores for each catalog item, yielding a recommendation ranking. At a high level, GenRec: Verbalizes user histories, item metadata, and context as text ## How DigitalOcean Served Kimi K3 on Day Zero DevFeed: [How DigitalOcean Served Kimi K3 on Day Zero]() Original publisher: [Read original article]() Author: Shree Murthy Published: 2026-07-30T17:10:40Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Digital Ocean](), [Inference](), [GPU](), [vllm](), [Hardware](), [NVLink](), [OpenAI]() Tags: [benchmarks](), [digitalocean](), [distributed](), [engineering](), [gpu](), [inference](), [nvidia](), [nvlink](), [reasoning](), [streaming](), [vllm]() ### AI overview DigitalOcean describes how it served the Kimi K3 model on its Inference Engine from day zero, including GPU selection, distributed serving with llm-d, vLLM tuning, and verification against Moonshot AI's benchmarks. ### Source excerpt DigitalOcean launched Kimi K3 on day 0. It's already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a model this size running well on day zero took real work across several teams. Thanks to Moonshot AI, Inferact, RadixArk, NVIDIA, and AMD for the help getting there. Standing up a new model, integrating it into DigitalOcean's Inference Engine, and showcasing its unique attributes on day 0 takes three things: the right hardware, a tuned serving stack, and rigorous verification against Moonshot's own benchmarks. Here are the lessons we learned along the way: Hardware selection and implementation We selected NVIDIA HGX™ B300 and AMD Instinct™ MI350x GPUs to run K3 because these instances provide the memory capacity, FLOPs, and interconnect horsepower necessary for a model of K3's size and architecture. We built our distributed inference stack with llm-d because it includes native support for GPU type heterogeneity. This let us quickly onboard K3 to both AMD and NVIDIA platforms. Kimi K3 has roughly 2.78 trillion total parameters, 896 routed experts, and an attention stack that interleaves 69 Kimi Delta Attention (KDA) layers with 24 Gated Multi-head Latent Attention (MLA) layers. Kimi-K3 weights are ~1.56 TB in total, which requires about 195 GiB per GPU. Given such a large memory footprint for the weights alone, and a need to keep enough headroom for KV cache and activations, the practical unit of deployment is an 8x NVIDIA HGX B300 or AMD Instinct MI350X server. Both have 288GB of VRAM capacity, and after loading the weights, there is still some amount of practical memory left for the KV cache. Entire weights cannot be loaded on a single GPU. That's where the high-speed scaled-up NVIDIA's NVLink or AMD's Infinity Fabric is critical to ensure there is enough interconnect horsepower for bandwidth intensive, latency sensitive attention and expert parallel computations. Model ## Инференс LLM: от KV-кэша до продакшен-деплоя DevFeed: [Инференс LLM: от KV-кэша до продакшен-деплоя]() Original publisher: [Read original article]() Author: a\_ryzhov (hh.ru, Конференции Олега Бунина (Онтико)) Published: 2026-07-27T05:30:45Z Content type: tutorial Language: ru Sources: [HeadHunter RU]() Topics: [Large Language Model](), [Caching](), [on-prem](), [genai](), [MLOps](), [sglang](), [vllm]() Tags: [2026](), [compute](), [genai](), [http](), [inference](), [kv-cache](), [llm](), [mlops](), [on-prem](), [sglang](), [sram](), [vllm]() ### AI overview This Russian-language developer article explains how LLM inference behaves in on-premises production environments in 2026. It argues that GPU memory management is the main efficiency constraint, describes how KV caching shifts decoding from compute-bound to memory-bandwidth-bound work, and introduces vLLM and SGLang as ways to address the problem. ### Source excerpt Привет! Я Саша Рыжов, MLOps-инженер в hh.ru, уже три года занимаюсь развитием инфраструктуры для искусственного интеллекта. Компании, которые развивают GenAI, рано или поздно приходят к задачам по запуску LLM на собственном железе. В статье я расскажу, как обстоят дела с движками инференса в 2026 году и как запустить on-prem-прод и не изобрести при этом велосипед. Читать далее ## Welcome Inkling by Thinking Machines DevFeed: [Welcome Inkling by Thinking Machines]() Original publisher: [Read original article]() Author: ben burtenshaw; merve; Pedro Cuenca; Aritra Roy Gosthipaty; Andres Marafioti Published: 2026-07-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Large Language Model](), [Mixture of Experts (MoE)](), [multimodal](), [hugging face](), [Inference](), [Deployment](), [Fine-tuning](), [Open Source](), [sglang]() Tags: [announcement](), [artificial-intelligence](), [audio](), [context-window](), [deployment](), [fine-tuning](), [generation](), [inference](), [inference-endpoints](), [llama-cpp](), [llms](), [mixture-of-experts](), [mixture-of-experts-moe](), [moe](), [multimodal](), [open-source](), [sglang](), [transformers](), [vllm]() ### AI overview Thinking Machines Lab's Inkling is presented as a large open multimodal language model that accepts image, text, and audio inputs. The article covers its mixture-of-experts architecture, million-token context window, reasoning across modalities, fine-tuning use cases, model variants, and deployment through Hugging Face Inference Endpoints and inference frameworks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. [Next page]()