# tgi

A Hugging Face toolkit for deploying and serving large language models with high-performance text generation.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## FORGE: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly

DevFeed: [FORGE: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly](<https://devfeed.tech/articles/forge-fault-oriented-rework-code-generation-for-executable-interlocking-assembly-115325.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02536>)

Author: Fuqiang Wang, Jiahao Fu, Ren Hefeng, Yining Qian, Chuanting Zhang, Cheng Tan, Yuwen Liu, Jingxuan Wei

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Code generation](<https://devfeed.tech/topics/code-generation.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [essential](<https://devfeed.tech/tags/essential.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [forge](<https://devfeed.tech/tags/forge.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [rework](<https://devfeed.tech/tags/rework.md>)

### AI overview

FORGE addresses the gap between identifying an assembly fault and generating executable rework that restores the intended configuration. Its FORGE-Bench benchmark contains 4,000 tasks across 26 interlocking assemblies, with five assemblies held out to evaluate generalization to unseen structures. The framework separates rework into two stages: Rework Structuring captures recovery dependencies caused by faults, and Action Grounding maps them to state-conditioned executable actions. The reported experiments find that locally plausible repair code can still fail, and that linking structural recovery reasoning to executable actions is important for verified restoration.

### Source excerpt

arXiv:2611.02536v1 Announce Type: new Abstract: Large language models have demonstrated strong capabilities in structured reasoning and code generation, yet identifying an assembly fault does not guarantee the ability to generate executable rework programs. In interlocking assemblies, a locally plausible repair program may violate blocking dependencies, omit required recovery operations, or terminate in an incorrect configuration. We identify this challenge as fault-conditioned executable rework generation, where models must jointly reason about structural failures and generate programs that restore constrained assembly states. To benchmark this capability, we introduce FORGE-Bench, containing 4,000 tasks across 26 interlocking assemblies with five assemblies held out for evaluating generalization to unseen structures. We further propose FORGE (Fault-Oriented Rework Code Generation for Executable Interlocking Assembly), a structured state-to-program framework that decomposes rework generation into Rework Structuring and Action Grounding. Rework Structuring captures fault-induced recovery dependencies, while Action Grounding maps these dependencies into state-conditioned executable actions. Experiments show that locally reasonable repair code is insufficient for reliable assembly recovery, and that connecting structural recovery reasoning with executable action realization is essential for verified target restoration.

## ReFrame: Silence-Aware Compute Reclamation for Full-Duplex Speech Models

DevFeed: [ReFrame: Silence-Aware Compute Reclamation for Full-Duplex Speech Models](<https://devfeed.tech/articles/reframe-silence-aware-compute-reclamation-for-full-duplex-speech-models-114612.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01835>)

Author: Zheng Zhou, Guanyi Chen, Jing Wei, Haoyuan Liang, Jian Zhang, Haohuan Fu

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [speech-to-speech](<https://devfeed.tech/topics/speech-to-speech.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [compute](<https://devfeed.tech/tags/compute.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [model-inference](<https://devfeed.tech/tags/model-inference.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [speech](<https://devfeed.tech/tags/speech.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

ReFrame reclaims compute in full-duplex speech inference by using dialogue state to schedule work. During mutual silence, Mutual Silence Speculation (MSS) evaluates multiple Temporal continuations and verifies stability before correcting Mimi audio tokens in batches. While the user speaks and the agent listens, Listening Frame Folding (LFF) processes arrived frames in parallel and accepts only a verified prefix. Frame-Budget Reinvestment (FBR) uses released intervals for resumable retrieval and language-model workers, then supplies references through Moshi's conditioning interface. Across 1,225 FDBench conversations on five models, the paper reports 18.80-31.01% lower measured GPU execution time with 96.92-100% non-PAD text agreement. In HaluEval tests, FBR raised correct spoken answers from 86 to 205 under the same auxiliary-service budget.

### Source excerpt

arXiv:2611.01835v1 Announce Type: new Abstract: Full-duplex speech models execute dense model inference for every audio frame throughout a live interaction. This uniform framewise schedule creates sustained GPU load across dialogue states whose computation has strong, reusable temporal structure. ReFrame formulates full-duplex inference as a silence-aware computation problem. Mutual Silence Speculation (MSS) operates when both participants are silent: it evaluates multi-frame Temporal continuations, verifies text and hidden-state stability, and applies batched Mimi audio-token correction. Listening Frame Folding (LFF) operates while the user speaks and the agent listens: it processes already-arrived user frames through parallel Temporal execution and frame-batched audio-token generation while accepting only the verified prefix. Frame-Budget Reinvestment (FBR) assigns the released frame intervals to resumable external retrieval and language-model workers, then injects timely references through Moshi's conditioning interface. We instantiate ReFrame on five full-duplex models spanning Moshi-family, phase-structured, and flattened autoregressive generation. Across the complete 1,225-conversation FDBench, ReFrame reduces measured GPU execution time by 18.80-31.01% and retains 96.92-100% non-PAD text agreement on five different models. On HaluEval tests, FBR increases the number of correct generated spoken answers from 86 to 205 under the same auxiliary-service budget. ReFrame turns recurring dialogue structure into lower inference cost and higher end-to-end service capability.

## DiffLTM: Continuous Diffusions Large Tabular Models for Conditional Generation

DevFeed: [DiffLTM: Continuous Diffusions Large Tabular Models for Conditional Generation](<https://devfeed.tech/articles/diffltm-continuous-diffusions-large-tabular-models-for-conditional-generation-96945.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30472>)

Author: Alankrita Bhatt, Mukur Gupta, Andrea Montanari, Pulkit Tandon

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [applications](<https://devfeed.tech/tags/applications.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [best](<https://devfeed.tech/tags/best.md>), [capability](<https://devfeed.tech/tags/capability.md>), [capture](<https://devfeed.tech/tags/capture.md>), [cardinality](<https://devfeed.tech/tags/cardinality.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [code](<https://devfeed.tech/tags/code.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cross-entropy](<https://devfeed.tech/tags/cross-entropy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [generation](<https://devfeed.tech/tags/generation.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [insights](<https://devfeed.tech/tags/insights.md>), [key](<https://devfeed.tech/tags/key.md>), [labor](<https://devfeed.tech/tags/labor.md>), [models](<https://devfeed.tech/tags/models.md>)

### AI overview

DiffLTM targets conditional generation for mixed numerical and categorical tabular data, where high-cardinality, imbalanced categories and rare subpopulations make accurate generation difficult. The paper proposes a continuous diffusion model, using cross-entropy loss for categorical entries and providing a theoretical justification for that choice. It also introduces metrics focused on rare subpopulations and extends conditional-generation techniques to support new types of conditioning. On labor data, the authors report that the model captures classical econometric insights. The supplied abstract gives no quantitative results or details for comparing performance with hybrid continuous and discrete diffusion models.

### Source excerpt

arXiv:2610.30472v1 Announce Type: new Abstract: Conditional generation is a key capability to be supported by Large Tabular Models, enabling a number of applications. Accurate conditional generation has to contend with two key challenges: $(i)$ Data comprise a mixture of numerical and categorical entries, the latter often with high cardinality and heavily unbalanced proportions; $(ii)$ Conditional generation becomes increasingly difficult for rare sub-populations. The best models in this class use a hybrid of continuous and discrete diffusions. In this paper, we design a continuous diffusion model with strong generation quality. An important element is the use of cross-entropy loss for categoricals, for which we provide a first theoretical justification. We introduce metrics that focus on rare subpopulations. Further, we generalize conditional generation techniques to enable new types of conditioning. Finally, we show that our model (trained on labor data) can capture classical insights from econometrics. Code is available at https://github.com/granica-ai/DiffLTM

## Jev: Bounded semantic judgments for messy text

DevFeed: [Jev: Bounded semantic judgments for messy text](<https://devfeed.tech/articles/jev-bounded-semantic-judgments-for-messy-text-90645.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/10/20/jev-bounded-semantic-judgments-messy-text>)

Author: Maarten Vandeperre

Published: 2026-10-20T13:02:39Z

Content type: article

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [generative-ai-tools](<https://devfeed.tech/topics/generative-ai-tools.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [redhat ubi](<https://devfeed.tech/topics/redhat-ubi.md>), [json lines format](<https://devfeed.tech/topics/json-lines-format.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Discord](<https://devfeed.tech/topics/discord.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-model](<https://devfeed.tech/tags/ai-model.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-camel](<https://devfeed.tech/tags/apache-camel.md>), [api](<https://devfeed.tech/tags/api.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [code](<https://devfeed.tech/tags/code.md>), [container](<https://devfeed.tech/tags/container.md>), [red-hat](<https://devfeed.tech/tags/red-hat.md>)

### AI overview

The article explains how Jev, a managed AI service from TypeSafe, makes bounded semantic judgments from application state using choice, yes-or-no probability, and score questions. It discusses confidence, calibration, thresholds, human review, and comparisons with generative LLMs and conventional classifiers, then describes Jev's hosted-only deployment limits and contrasts it with self-hosted inference options.

### Source excerpt

Imagine that you are building the software for a coffee shop. A customer types: "One big coffee with lots of milk, but no normal milk, please." A barista might guess that the customer means a large latte with plant-based milk. The application needs a more careful process. "Big" probably means large, but "no normal milk" does not tell us whether the customer wants oat, soy, or almond milk. An intelligent component can identify the likely intent. The post Jev: Bounded semantic judgments for messy text appeared first on Red Hat Developer.

## StepFun Releases Step 5 Preview Open Weights: 600B MoE, 27B Active, 1M Context

DevFeed: [StepFun Releases Step 5 Preview Open Weights: 600B MoE, 27B Active, 1M Context](<https://devfeed.tech/articles/stepfun-releases-step-5-preview-open-weights-600b-moe-27b-active-1m-context-89245.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/10/16/stepfun-releases-step-5-preview-open-weights-600b-moe-27b-active-1m-context/>)

Author: Michal Sutter

Published: 2026-10-17T06:29:55Z

Content type: news

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [moe](<https://devfeed.tech/tags/moe.md>), [new-releases](<https://devfeed.tech/tags/new-releases.md>), [open-weights](<https://devfeed.tech/tags/open-weights.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [staff](<https://devfeed.tech/tags/staff.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

StepFun released Step 5 Preview as an open-weight sparse Mixture-of-Experts model with 600 billion total parameters, 27 billion active per token, and up to a one-million-token context window. The article describes its coding and agentic uses, benchmark results, substantial data-center GPU requirements, and API pricing.

### Source excerpt

StepFun has released Step 5 Preview as open weights. FP8 and BF16 checkpoints are live on Hugging Face and ModelScope. Step 5 Preview is a sparse Mixture-of-Experts model from StepFun built for long-horizon agentic work. It has 600B total parameters, 27B active per token and up to 1M tokens of context. For developers, this is [...] The post StepFun Releases Step 5 Preview Open Weights: 600B MoE, 27B Active, 1M Context appeared first on MarkTechPost.

## Serving Reflection Beam on vLLM from Day 0

DevFeed: [Serving Reflection Beam on vLLM from Day 0](<https://devfeed.tech/articles/serving-reflection-beam-on-vllm-from-day-0-88770.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-16-reflection-beam>)

Author: Inferact, Reflection, and Nvidia

Published: 2026-10-16T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reflection](<https://devfeed.tech/tags/reflection.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

vLLM describes serving Reflection's Beam-1.0, a 501-billion-parameter mixture-of-experts model, and details its support for reasoning, tool calls, and structured outputs. The article explains model-specific kernel and serving optimizations, reporting 2.71x faster batch-one decoding and 1.87x faster prefill than the initial implementation. It also covers DFlash2 speculative decoding, adaptive verification, performance benchmarks, and setup options.

### Source excerpt

vLLM supports Reflection's Beam-1.0 (501B total, 23B active) from day 0, with agent-driven optimization that cut decode latency 2.71x and DFlash2 speculative decoding with adaptive verification.

## Muse Spark Is Now an Open-Weights Model

DevFeed: [Muse Spark Is Now an Open-Weights Model](<https://devfeed.tech/articles/muse-spark-is-now-an-open-weights-model-87992.md>)

Original publisher: [Read original article](<https://pytorch.org/blog/muse-spark-open/>)

Author: PyTorch Team at Meta

Published: 2026-10-15T17:55:52Z

Content type: release

Language: en

Sources: [PyTorch](<https://devfeed.tech/sources/blog-pytorch.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [blog](<https://devfeed.tech/tags/blog.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [meta](<https://devfeed.tech/tags/meta.md>), [muse-spark](<https://devfeed.tech/tags/muse-spark.md>), [open-weights](<https://devfeed.tech/tags/open-weights.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>)

### AI overview

Meta released Muse Spark Open, an open-weights model designed for coding and agentic work. The release includes portable Helion GPU kernels for inference and a TorchTitan model definition supporting supervised fine-tuning, LoRA, and reinforcement learning.

### Source excerpt

Today, Meta is releasing the weights for Muse Spark Open, the most capable US open-weights model to date. It's available today on Hugging Face under the Muse Model License. Muse... The post Muse Spark Is Now an Open-Weights Model appeared first on PyTorch.

## Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM

DevFeed: [Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM](<https://devfeed.tech/articles/announcing-day-0-support-for-nvidia-nemotron-3-5-super-vl-on-vllm-88077.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-10-15-nemotron-3-5-super-vl-vllm>)

Author: NVIDIA Nemotron Team

Published: 2026-10-15T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [batching](<https://devfeed.tech/tags/batching.md>), [caching](<https://devfeed.tech/tags/caching.md>), [day-0-support](<https://devfeed.tech/tags/day-0-support.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [image](<https://devfeed.tech/tags/image.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [model-support](<https://devfeed.tech/tags/model-support.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [nvidia-nemotron](<https://devfeed.tech/tags/nvidia-nemotron.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [token](<https://devfeed.tech/tags/token.md>), [video](<https://devfeed.tech/tags/video.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

NVIDIA announces support for Nemotron 3.5 Super VL in vLLM, describing a multimodal model for long-running agent workflows that can process text, images, documents, and video. The article explains serving options, including an OpenAI-compatible API, continuous batching, prefix caching, speculative decoding, and NVFP4 inference, and outlines customization with NVIDIA NeMo. It also reports performance comparisons for agentic tasks and points readers to model weights and a getting-started cookbook.

### Source excerpt

Run NVIDIA Nemotron 3.5 Super VL on vLLM for multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.

## Top Tools for Serving Open-Source Models on Kubernetes

DevFeed: [Top Tools for Serving Open-Source Models on Kubernetes](<https://devfeed.tech/articles/top-tools-for-serving-open-source-models-on-kubernetes-86942.md>)

Original publisher: [Read original article](<https://hackernoon.com/top-tools-for-serving-open-source-models-on-kubernetes?source=rss>)

Author: SIEmon

Published: 2026-10-14T10:32:32Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [good-company](<https://devfeed.tech/tags/good-company.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-inference](<https://devfeed.tech/tags/gpu-inference.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [llms](<https://devfeed.tech/tags/llms.md>), [nvidia-dynamo](<https://devfeed.tech/tags/nvidia-dynamo.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [sglang-deployment](<https://devfeed.tech/tags/sglang-deployment.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This guide compares the components needed to serve open-source language models on Kubernetes, including runtimes, deployment platforms, request routing, and GPU autoscaling. It discusses workload trade-offs and the cold-start costs of scaling GPU pods to zero.

### Source excerpt

Explore the best open-source tools for serving LLMs on Kubernetes, from vLLM and SGLang to KServe, llm-d, NVIDIA Dynamo, SIE, and GPU autoscaling.

## From Activity to Intent: Generating User Journeys with LLMs

DevFeed: [From Activity to Intent: Generating User Journeys with LLMs](<https://devfeed.tech/articles/from-activity-to-intent-generating-user-journeys-with-llms-84247.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/from-activity-to-intent-generating-user-journeys-with-llms-9652b9417626?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-10-09T15:01:02Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>)

Tags: [engineering](<https://devfeed.tech/tags/engineering.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llms](<https://devfeed.tech/tags/llms.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [user-experience](<https://devfeed.tech/tags/user-experience.md>)

### AI overview

Pinterest Engineering describes replacing a staged keyword clustering pipeline with a fine-tuned 4B Qwen3 large language model that infers and ranks user journeys from chronological activity logs. It covers synthetic training data and LoRA fine-tuning, production serving with NVIDIA Dynamo and vLLM, safety filters, evaluation, and experiments with semantic IDs and user feedback.

### Source excerpt

Lin Zhu | Sr. Staff Machine Learning Engineer; Manan Kalra | Machine Learning Engineer II; Logan Jeon | Sr. Machine Learning Engineer; Ye Liu | Staff Machine Learning Engineer; Xiangyi Chen | Sr. Machine Learning Engineer; Jaewon Yang | Principal Machine Learning Engineer; Jinwen Xu | Manager II, Machine Learning Engineering; Tingting Zhu | Sr. Manager, Engineering; Sudarshan Lamkhede | Director, Machine Learning Engineering Pinterest is built to get inspired and then turn the inspiration into realization -- a dinner, a renovation, a wedding, a new skill. That only works if we understand more than the Pin a user just clicked. The useful unit of personalization isn't the thing someone just clicked, it's what they're trying to accomplish. Such goals unfold over weeks and months, across dozens of sessions, through hundreds of saves. We call those goals journeys: the intersection of a user's interests, intent, and context at a point in time. A journey is a sequence of interactions, often spanning sessions, centered on one interest and a clear intent. For example, "Summer dresses" are an interest. "Figuring out what's in style for a summer wedding I'm attending in August" is a journey. Someone might be interested in five at once: a kitchen remodel, a vegetable garden, learning sourdough and the mix changes as life does. If we can name those journeys, recommendations and notifications can help someone finish a project instead of bouncing between unrelated interests. This post is about how we infer them in production: a fine-tuned 4-billion-parameter large language model (LLM) that reads a chronological activity log and emits a ranked list of journey names, in the user's locale. We didn't start there. The initial user journey system clustered keywords and attached attributes with separate machine learning models. This post is the follow-up which covers how we formulate journey inference as generation, how we distill frontier-model quality into a 4B student we can actually s

## High Performance by Default: Why Speed Shouldn't Be a Setup Step

DevFeed: [High Performance by Default: Why Speed Shouldn't Be a Setup Step](<https://devfeed.tech/articles/high-performance-by-default-why-speed-shouldn-t-be-a-setup-step-84270.md>)

Original publisher: [Read original article](<https://www.fastly.com/blog/high-performance-by-default-why-speed-shouldnt-setup-step/>)

Author: Sarah Sang; Delen Trance

Published: 2026-10-07T00:00:00Z

Content type: release

Language: en

Sources: [Fastly Blog](<https://devfeed.tech/sources/fastly-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Core Web Vitals](<https://devfeed.tech/topics/core-web-vitals.md>)

Tags: [chrome-user-experience-report](<https://devfeed.tech/tags/chrome-user-experience-report.md>), [compression](<https://devfeed.tech/tags/compression.md>), [core-web-vitals](<https://devfeed.tech/tags/core-web-vitals.md>), [fastly](<https://devfeed.tech/tags/fastly.md>), [http](<https://devfeed.tech/tags/http.md>), [http-3](<https://devfeed.tech/tags/http-3.md>), [performance](<https://devfeed.tech/tags/performance.md>), [tls](<https://devfeed.tech/tags/tls.md>)

### AI overview

Fastly introduces Recommended Defaults for new services, automatically enabling compression with a Gzip fallback, HTTP/3, and TLS with HSTS. Settings remain visible and editable before launch, giving teams a performance-focused starting configuration without removing their control.

### Source excerpt

Discover how Fastly's Recommended Defaults automatically optimize performance with built-in compression, HTTP/3, and TLS so speed is never a setup step.

## Serving LLM Inference with NVIDIA Triton and Eleuther AI

DevFeed: [Serving LLM Inference with NVIDIA Triton and Eleuther AI](<https://devfeed.tech/articles/serving-llm-inference-with-nvidia-triton-and-eleuther-ai-78029.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/serving-inference-for-llms-nvidia-triton-inference-server-eleuther-ai>)

Author: CoreWeave

Published: 2026-10-02T14:12:41Z

Content type: article

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [auto-scaling](<https://devfeed.tech/tags/auto-scaling.md>), [cloud-infrastructure](<https://devfeed.tech/tags/cloud-infrastructure.md>), [content-type](<https://devfeed.tech/tags/content-type.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scale](<https://devfeed.tech/tags/scale.md>), [triton-inference-server](<https://devfeed.tech/tags/triton-inference-server.md>)

### AI overview

The article explains how NVIDIA Triton Inference Server with FasterTransformer and CoreWeave's GPU infrastructure can serve large language model inference. It describes multi-GPU execution, autoscaling, and benchmarks reporting 30-40% faster inference than a vanilla Hugging Face PyTorch implementation for the tested GPT models.

### Source excerpt

Discover how Eleuther AI leveraged NVIDIA Triton Inference Server and CoreWeave's infrastructure to efficiently serve LLM inference at scale, optimizing performance, speed, and resource utilization.

## The Evolution of Large Language Models and How They Stack Up

DevFeed: [The Evolution of Large Language Models and How They Stack Up](<https://devfeed.tech/articles/the-evolution-of-large-language-models-and-how-they-stack-up-78040.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/the-evolution-of-large-language-models-and-how-they-stack-up>)

Author: CoreWeave

Published: 2026-10-02T14:12:41Z

Content type: article

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Self Hosted AI Tools](<https://devfeed.tech/topics/self-hosted-ai-tools.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [coreweave](<https://devfeed.tech/tags/coreweave.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>)

### AI overview

The article discusses large language models and the infrastructure for training, fine-tuning, and serving them. It compares model options for developers and highlights BLOOM 176B as an open model supporting 46 human languages and 13 programming languages.

### Source excerpt

The rise of new large (and mega) language models with seemingly limitless applications are poised to change the world as we know it. As LLM parameters evolve into the trillions, CoreWeave is diving deeper into the most promising and pragmatic models, offering advice to entrepreneurs, developers and researchers on the latest options to build and deploy NLP focused products and services as quickly and cost effectively as possible.

## From No-Code to Pro: Optimizing Mistral-7B on Runpod for Power Users

DevFeed: [From No-Code to Pro: Optimizing Mistral-7B on Runpod for Power Users](<https://devfeed.tech/articles/from-no-code-to-pro-optimizing-mistral-7b-on-runpod-for-power-users-78096.md>)

Original publisher: [Read original article](<https://www.runpod.io/blog/from-no-code-to-pro-optimizing-mistral-7b-on-runpod-for-power-users>)

Author: Eliot Cowley

Published: 2026-09-26T14:29:16Z

Content type: tutorial

Language: en

Sources: [Runpod Blog.](<https://devfeed.tech/sources/runpod-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cost](<https://devfeed.tech/tags/cost.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [docker](<https://devfeed.tech/tags/docker.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [performance](<https://devfeed.tech/tags/performance.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

This tutorial builds on a no-code Mistral-7B deployment by showing how to serve the model on Runpod with quantized GGUF models and vLLM workers. It explains deployment through pods and serverless endpoints, and compares GPU performance, inference speed, and cost.

### Source excerpt

Optimize Mistral-7B deployment with Runpod by using quantized GGUF models and vLLM workers, compare GPU performance across pods and serverless endpoints.

## Set Up a Chatbot with Oobabooga on Runpod

DevFeed: [Set Up a Chatbot with Oobabooga on Runpod](<https://devfeed.tech/articles/set-up-a-chatbot-with-oobabooga-on-runpod-78123.md>)

Original publisher: [Read original article](<https://www.runpod.io/blog/oobabooga-chatbot-runpod>)

Author: Pardeep Singh

Published: 2026-09-25T21:48:07Z

Content type: tutorial

Language: en

Sources: [Runpod Blog.](<https://devfeed.tech/sources/runpod-blog.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [chatbot](<https://devfeed.tech/tags/chatbot.md>), [huggingface](<https://devfeed.tech/tags/huggingface.md>), [text-generation](<https://devfeed.tech/tags/text-generation.md>)

### AI overview

A tutorial for deploying Oobabooga's Text Generation WebUI on Runpod with the Pygmalion 6B chatbot model. It explains how to load character profiles, interact with the chatbot, and switch to generic text generation with another model such as GPT-J-6B.

### Source excerpt

This tutorial walks you through deploying Oobabooga's Text Generation WebUI using the Runpod template. Includes steps for loading Pygmalion 6B and.

## vllm-metal Brings Batched Inference and Concurrent Serving to Apple Silicon

DevFeed: [vllm-metal Brings Batched Inference and Concurrent Serving to Apple Silicon](<https://devfeed.tech/articles/announcing-vllm-metal-concurrent-serving-on-apple-silicon-79491.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-09-22-vllm-metal-v0-28-0>)

Author: Ranran Haoran Zhang, Lik Xun Yuan, Chao Ju Chen, Eric Curtin, Michael Goin

Published: 2026-09-22T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [MLX](<https://devfeed.tech/topics/mlx.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Structured concurrency](<https://devfeed.tech/topics/structured-concurrency.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [apple](<https://devfeed.tech/tags/apple.md>), [apple-silicon](<https://devfeed.tech/tags/apple-silicon.md>), [batching](<https://devfeed.tech/tags/batching.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [metal](<https://devfeed.tech/tags/metal.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [mtp](<https://devfeed.tech/tags/mtp.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The vllm-metal release brings vLLM's scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, using MLX and Metal for execution. The document describes batched multi-token prediction, hybrid-model support, memory controls, and benchmark comparisons of concurrent serving workloads. It also outlines installation and reproduction steps for serving benchmarks.

### Source excerpt

vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.

## Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking

DevFeed: [Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking](<https://devfeed.tech/articles/introducing-gemini-3-8-live-and-3-8-live-extended-thinking-89441.md>)

Original publisher: [Read original article](<https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-live-gemini-3-8-live-extended-thinking/>)

Author: Malini Jaganathan

Published: 2026-09-15T17:00:00Z

Content type: release

Language: en

Sources: [Gemini Models](<https://devfeed.tech/sources/gemini-models.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [synthid](<https://devfeed.tech/topics/synthid.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-products](<https://devfeed.tech/tags/ai-products.md>), [ai-studio](<https://devfeed.tech/tags/ai-studio.md>), [api](<https://devfeed.tech/tags/api.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gemini-3-8-live](<https://devfeed.tech/tags/gemini-3-8-live.md>), [gemini-api](<https://devfeed.tech/tags/gemini-api.md>), [gemini-enterprise-agent-platform](<https://devfeed.tech/tags/gemini-enterprise-agent-platform.md>), [gemini-models](<https://devfeed.tech/tags/gemini-models.md>), [google](<https://devfeed.tech/tags/google.md>), [google-ai](<https://devfeed.tech/tags/google-ai.md>), [none](<https://devfeed.tech/tags/none.md>), [thinking](<https://devfeed.tech/tags/thinking.md>)

### AI overview

Google introduces Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, models designed for real-time voice interactions. The announcement describes visual input processing, background tool and API calls, multilingual conversation, and extended reasoning for complex workflows. Developers can access the models through the Gemini API and Google AI Studio, with the Live API also enabling platforms to build voice interfaces.

### Source excerpt

Text "Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking" with the Gemini Spark, all on a light blue background

## Deploying Kimi K3 in 4-bit on a single 8xB300 Pod on Runpod

DevFeed: [Deploying Kimi K3 in 4-bit on a single 8xB300 Pod on Runpod](<https://devfeed.tech/articles/deploying-kimi-k3-in-4-bit-on-a-single-8xb300-pod-on-runpod-78089.md>)

Original publisher: [Read original article](<https://www.runpod.io/blog/deploying-kimi-k3-in-4-bit-on-a-single-8xb300-pod-on-runpod>)

Author: Brendan McKeag

Published: 2026-09-13T15:06:30Z

Content type: tutorial

Language: en

Sources: [Runpod Blog.](<https://devfeed.tech/sources/runpod-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>)

Tags: [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [ai](<https://devfeed.tech/tags/ai.md>), [api](<https://devfeed.tech/tags/api.md>), [batching](<https://devfeed.tech/tags/batching.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cuda-graphs](<https://devfeed.tech/tags/cuda-graphs.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>)

### AI overview

This guide explains how to deploy Kimi K3 on a single Runpod 8xB300 pod using vLLM and CUDA 13. It covers downloading the model weights, configuring an OpenAI-compatible endpoint, and the limits of fitting a 128K context window alongside concurrent requests. The article also compares a self-managed pod with Runpod's public endpoint, noting that the endpoint may suit spiky or moderate workloads while a pod can fit sustained, high-volume use.

### Source excerpt

How to get up and running with Kimi K3 without the overhead of running an entire cluster.

## Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding

DevFeed: [Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding](<https://devfeed.tech/articles/parallel-all-the-way-down-beyond-single-token-generation-with-speculative-decoding-79467.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-07-28-speculators-parallel-drafting>)

Author: Alexandre Marques, Megan Flynn, Helen Zhao, Krishna Teja Chitty Venkata, Chibueze Ukachi (Red Hat AI)

Published: 2026-07-28T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [dflash](<https://devfeed.tech/tags/dflash.md>), [dspark](<https://devfeed.tech/tags/dspark.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [peagle](<https://devfeed.tech/tags/peagle.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [speculators](<https://devfeed.tech/tags/speculators.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

Speculators and vLLM support three parallel drafting algorithms for speculative decoding in large language model serving: P-EAGLE, DFlash, and DSpark. The article explains how these methods generate draft tokens in parallel, describes their distinct training and inference approaches, and presents them as ways to reduce drafting latency and improve serving throughput. It notes that performance varies by model, task, and hardware, and that the article's benchmark plots were corrected after an erroneous environment setup.

### Source excerpt

Speculators and vLLM now support P-EAGLE, DFlash, and DSpark -- three parallel drafting algorithms that move beyond sequential token generation to deliver faster, simpler, and more scalable speculative decoding for LLM serving.

## Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving

DevFeed: [Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving](<https://devfeed.tech/articles/announcing-vllm-afd-plugin-disaggregating-attention-and-ffn-for-flexible-moe-serving-79465.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-07-23-vllm-afd-plugin>)

Author: AFD Plugin Contributors

Published: 2026-07-23T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [attention](<https://devfeed.tech/tags/attention.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [moe](<https://devfeed.tech/tags/moe.md>), [npu](<https://devfeed.tech/tags/npu.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The vLLM AFD Plugin is an experimental plugin that separates Attention and FFN into independently deployed services for MoE inference. It retains vLLM's request lifecycle and OpenAI-compatible serving interface, with GPU and Ascend NPU backends and synchronous and asynchronous connectors. Benchmarks show throughput varies by Attention-to-FFN allocation: one tested configuration exceeded the EP64 baseline, while another fell below it. The reported results are limited experiments, not general production performance claims.

### Source excerpt

What vLLM AFD Plugin adds to the vLLM ecosystem: Attention-FFN disaggregation for MoE serving, GPU and Ascend NPU backends, connector-based execution, and graph and ubatching support.

## EAGLE3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD Quark

DevFeed: [EAGLE3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD Quark](<https://devfeed.tech/articles/eagle3-speculative-decoding-on-amd-instinct-gpus-training-and-serving-with-vllm-and-amd-quark-79458.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-07-13-eagle-3-amd-instinct>)

Author: Larry Li, Chao Li, Haichen Zhang, Chun Fang, Andy Luo, Spandan Tiwari, and Ashish Sirasao

Published: 2026-07-13T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article explains how AMD Quark and vLLM are used to train, quantize, and serve EAGLE3 speculative-decoding drafts on AMD Instinct GPUs. It reports throughput gains of up to 2.00x for Kimi-K2.5 and 1.79x for MiniMax-M2.5 in specified 1K/1K benchmark sweeps, while noting that the random-prompt tests are throughput microbenchmarks rather than application-level workload benchmarks.

### Source excerpt

How AMD Quark trains, quantizes, and serves EAGLE3 speculative-decoding drafts with vLLM on AMD Instinct GPUs, delivering up to 2.00x throughput gains for Kimi-K2.5 and 1.79x for MiniMax-M2.5.

## vLLM x HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan

DevFeed: [vLLM x HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan](<https://devfeed.tech/articles/vllm-hpc-ops-high-performance-attention-and-moe-backends-from-tencent-hunyuan-79456.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-07-06-vllm-hpc-ops>)

Author: Tencent Hunyuan AI Infra Team and vLLM Team

Published: 2026-07-06T00:00:00Z

Content type: article

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [attention](<https://devfeed.tech/tags/attention.md>), [batching](<https://devfeed.tech/tags/batching.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [hopper](<https://devfeed.tech/tags/hopper.md>), [hpc](<https://devfeed.tech/tags/hpc.md>), [hpc-ops](<https://devfeed.tech/tags/hpc-ops.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [moe](<https://devfeed.tech/tags/moe.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

HPC-Ops attention and FP8 MoE kernels have been integrated into vLLM as native backends, targeting dynamic, mixed-length LLM serving workloads. The article describes load-balanced attention scheduling and a fused MoE pipeline, with reported benchmarks showing up to 2.95x faster mixed-length decode than a static schedule and average end-to-end reductions of about 24% in TTFT and 17% in TPOT on Hy3 across eight NVIDIA H20 GPUs.

### Source excerpt

How HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for Tencent Hunyuan Hy3, improving mixed-length decode, MoE latency, TTFT, and TPOT on NVIDIA H20.

## Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI

DevFeed: [Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI](<https://devfeed.tech/articles/fast-efficient-llm-inference-with-vllm-a-new-course-with-deeplearning-ai-79446.md>)

Original publisher: [Read original article](<https://vllm.ai/blog/2026-06-03-deeplearning-ai-vllm-course>)

Author: Cedric Clyburn

Published: 2026-06-03T00:00:00Z

Content type: release

Language: en

Sources: [vLLM Blog](<https://devfeed.tech/sources/vllm-blog.md>)

Topics: [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [tgi](<https://devfeed.tech/topics/tgi.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [community](<https://devfeed.tech/tags/community.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [inference](<https://devfeed.tech/tags/inference.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>), [open-source-ai](<https://devfeed.tech/tags/open-source-ai.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tradeoffs](<https://devfeed.tech/tags/tradeoffs.md>), [vllm-blog](<https://devfeed.tech/tags/vllm-blog.md>)

### AI overview

The article announces a free DeepLearning.AI course, developed with Red Hat, that teaches the LLM inference deployment lifecycle: compressing models, serving them with vLLM, and benchmarking latency, throughput, and quality. Hands-on labs cover quantization with LLM Compressor, vLLM serving features such as continuous batching and prefix caching, and load testing with GuideLLM.

### Source excerpt

What the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.

## How Together AI built the world's fastest speech-to-text stack

DevFeed: [How Together AI built the world's fastest speech-to-text stack](<https://devfeed.tech/articles/how-together-ai-built-the-world-s-fastest-speech-to-text-stack-80187.md>)

Original publisher: [Read original article](<https://www.together.ai/blog/how-together-ai-built-the-worlds-fastest-speech-to-text-stack>)

Author: Sebastien Beurnier

Published: 2026-05-29T00:00:00Z

Content type: article

Language: en

Sources: [Together.ai](<https://devfeed.tech/sources/together-ai.md>)

Topics: [asr](<https://devfeed.tech/topics/asr.md>), [Integration testing](<https://devfeed.tech/topics/integration-testing.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Service fabric](<https://devfeed.tech/topics/service-fabric.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [compile](<https://devfeed.tech/tags/compile.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cuda-graphs](<https://devfeed.tech/tags/cuda-graphs.md>), [domain-sockets](<https://devfeed.tech/tags/domain-sockets.md>), [epoll](<https://devfeed.tech/tags/epoll.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [speech](<https://devfeed.tech/tags/speech.md>), [speech-to-text](<https://devfeed.tech/tags/speech-to-text.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

Together describes how it optimized production speech-to-text serving across the full system, including profile-tuned TensorRT encoder execution, GPU-side decoder control flow, lower-copy audio processing, evented streaming I/O, and Python garbage collection. The article says these changes improved throughput and latency, including a 2-3x faster decoder and the removal of periodic p95 latency spikes.

### Source excerpt

Together AI built the fastest speech-to-text stack on Artificial Analysis by treating ASR as a full-path systems problem, not just a GPU inference problem.

[Next page](<https://devfeed.tech/topics/tgi.md?cursor=WyIyMDI2LTA1LTI5VDAwOjAwOjAwKzAwOjAwIiwgImZjZWIwOTg4LTNiZmMtNGQyNy1iNmRjLTE3MDk5MDQxNTc2YyJd>)