# kv-cache Published articles for kv-cache. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Инференс LLM: от KV-кэша до продакшен-деплоя DevFeed: [Инференс LLM: от KV-кэша до продакшен-деплоя]() Original publisher: [Read original article]() Author: a\_ryzhov (hh.ru, Конференции Олега Бунина (Онтико)) Published: 2026-07-27T05:30:45Z Content type: tutorial Language: ru Sources: [HeadHunter RU]() Topics: [Large Language Model](), [Caching](), [on-prem](), [genai](), [MLOps](), [sglang](), [vllm]() Tags: [2026](), [compute](), [genai](), [http](), [inference](), [kv-cache](), [llm](), [mlops](), [on-prem](), [sglang](), [sram](), [vllm]() ### AI overview This Russian-language developer article explains how LLM inference behaves in on-premises production environments in 2026. It argues that GPU memory management is the main efficiency constraint, describes how KV caching shifts decoding from compute-bound to memory-bandwidth-bound work, and introduces vLLM and SGLang as ways to address the problem. ### Source excerpt Привет! Я Саша Рыжов, MLOps-инженер в hh.ru, уже три года занимаюсь развитием инфраструктуры для искусственного интеллекта. Компании, которые развивают GenAI, рано или поздно приходят к задачам по запуску LLM на собственном железе. В статье я расскажу, как обстоят дела с движками инференса в 2026 году и как запустить on-prem-прод и не изобрести при этом велосипед. Читать далее ## Operational Costs and Performance Constraints of Self-Hosting Llama-3 DevFeed: [Operational Costs and Performance Constraints of Self-Hosting Llama-3]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-21T00:00:00Z Content type: opinion Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [llama](), [Inference](), [GPU](), [Large Language Model](), [Latency]() Tags: [cuda](), [gpu](), [inference](), [kv-cache](), [latency](), [llama](), [llm](), [operations](), [vram]() ### AI overview This opinion article argues that the total cost of self-hosting Llama-3 includes VRAM for model weights and per-user KV caches, memory-bandwidth limits, batching and latency trade-offs, idle GPU time, and operational staffing. It uses a Llama-3 70B example to illustrate why API savings may be offset by infrastructure and support costs. ### Source excerpt Llama-3 70B at 16-bit precision needs roughly 140 GB of VRAM just to sit idle, before the KV cache that every concurrent user requires. Inference is memory-bandwidth-bound rather than compute-bound: an A100's roughly 2000 GB/s against a 140 GB weight set caps single-stream generation near 14 tokens per second before batching. Add idle GPU time, batching-versus-latency trade-offs, and an ML infra engineer on pager duty, and that salary alone can erase the API savings. ## Instrumenting AI Agents: Why the Apology Metric Is a First Class Reliability Signal DevFeed: [Instrumenting AI Agents: Why the Apology Metric Is a First Class Reliability Signal]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-21T00:00:00Z Content type: article Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Artificial Intelligence](), [AI Agent](), [reliability](), [agent observability](), [context]() Tags: [ai](), [ai-agents](), [api](), [backend](), [context](), [dependency](), [distributed-systems](), [inference](), [json](), [kv-cache](), [latency](), [llm](), [llmops](), [metric](), [observability](), [orchestration](), [rag](), [reliability](), [reliability-engineering](), [signal](), [timeout]() ### AI overview This article argues that AI agents can return HTTP 200 responses while still failing business tasks. It proposes tracking apology responses as a reliability signal to expose context starvation caused by stalled dependencies, timeouts, stale synchronization, dropped data, or payload truncation in retrieval and orchestration pipelines. ### Source excerpt An AI agent that returns "I apologize, but I cannot find that order" ships an HTTP 200. Infrastructure dashboards call that healthy; the business calls it a failure. Tracking apology phrases as a first-class SLO surfaces context starvation, a stalled dependency, a timeout guard tripping, or silent payload truncation, none of which HTTP status codes and CPU graphs can see. ## Why MCP and Skills Are Complementary for AI Agents DevFeed: [Why MCP and Skills Are Complementary for AI Agents]() Original publisher: [Read original article]() Author: Michael Ryaboy Published: 2026-01-26T00:00:00Z Content type: article Language: en Sources: [Mintlify Blog]() Topics: [Model Context Protocol](), [AI Agent](), [Authentication](), [OAuth](), [Caching](), [context window](), [Claude](), [anthropic]() Tags: [ai](), [ai-agent](), [ai-trends](), [authentication](), [context-window](), [dev](), [kv-cache](), [mcp](), [oauth](), [performance](), [skills]() ### AI overview This article explains that MCP and Skills address different needs for AI agents: MCP provides standardized, authenticated and scoped access to external services, while Skills package the knowledge needed to use tools effectively. It also discusses MCP's context bloat and caching costs, and presents CLIs as an emerging complementary pattern. ### Source excerpt Skills and MCP solve different problems. Skills package knowledge. MCP packages authenticated, scoped access on a user's behalf. ## Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism DevFeed: [Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism]() Original publisher: [Read original article]() Author: Cen Zhao; Xiaodong Wang; Jianyu Huang Published: 2025-10-17T16:00:50Z Content type: article Language: en Sources: [Meta AI Research]() Topics: [Large Language Model](), [AI Inference](), [Inference Performance](), [LLM Techniques](), [GPU](), [sharding](), [long-context]() Tags: [ai-research](), [gpu](), [inference](), [kv-cache](), [large-language-models-llms](), [latency](), [llms](), [parallelism](), [performance](), [sharding]() ### AI overview Meta describes three forms of parallelism--tensor, context, and expert parallelism--for scaling large language model inference across GPUs. The article explains how prefill and decoding differ computationally and how these techniques target resource efficiency, throughput, and latency. ### Source excerpt At Meta, we are constantly pushing the boundaries of LLM inference systems to power applications such as the Meta AI App. We're sharing how we developed and implemented advanced parallelism techniques to optimize key performance metrics related to resource efficiency, throughput, and latency. The rapid evolution of large language models (LLMs) has ushered in a [...] Read More... The post Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism appeared first on Engineering at Meta.