# litellm

LiteLLM is an open-source AI gateway and Python SDK that provides a unified OpenAI-compatible interface for accessing multiple large language models, with routing, retries, authentication, cost tracking, and rate limiting.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## GPT-Live 1.1 now available on AI Gateway

DevFeed: [GPT-Live 1.1 now available on AI Gateway](<https://devfeed.tech/articles/gpt-live-1-1-now-available-on-ai-gateway-144669.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/gpt-live-1-1-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-19T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [OpenAI](<https://devfeed.tech/topics/openai.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>)

Tags: [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [application](<https://devfeed.tech/tags/application.md>), [audio](<https://devfeed.tech/tags/audio.md>), [bring-your-own-key](<https://devfeed.tech/tags/bring-your-own-key.md>), [byok](<https://devfeed.tech/tags/byok.md>), [cancellation](<https://devfeed.tech/tags/cancellation.md>), [channel](<https://devfeed.tech/tags/channel.md>), [client](<https://devfeed.tech/tags/client.md>), [commentary](<https://devfeed.tech/tags/commentary.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-retention](<https://devfeed.tech/tags/data-retention.md>), [delegation](<https://devfeed.tech/tags/delegation.md>), [event](<https://devfeed.tech/tags/event.md>), [failover](<https://devfeed.tech/tags/failover.md>), [gpt-6-2-sol](<https://devfeed.tech/tags/gpt-6-2-sol.md>), [gpt-live](<https://devfeed.tech/tags/gpt-live.md>), [inference](<https://devfeed.tech/tags/inference.md>), [openai](<https://devfeed.tech/tags/openai.md>)

### AI overview

GPT-Live 1.1 is a full-duplex voice model: callers can interrupt or add detail while it speaks. The source says it transcribes more accurately and maintains a more consistent voice and accent across languages than the previous GPT-Live model. Existing integrations can switch by changing the model ID while keeping the Live endpoint. For delegated work, the application enables client delegation, handles the delegation-created event, calls a text model such as openai/gpt-6.2-sol, and returns its result on the commentary channel for GPT-Live to speak. The application controls delegated permissions, confirmations, and cancellation. Delegated requests are billed separately, while voice-session usage continues during them. The example snippets omit connection setup and other streaming and event-handling details.

### Source excerpt

GPT-Live 1.1 from OpenAI is now available on AI Gateway. GPT-Live 1.1 is a full-duplex voice model, so it listens while it speaks and a caller can pause, interrupt, or add detail mid-reply. Compared with the previous GPT-Live model, it transcribes speech more accurately and keeps a more consistent accent and voice across languages. It uses the same Live endpoint and client delegation, so an existing GPT-Live integration only needs the new model ID. Simple example without delegation Run openai/gpt-live-1.1 by itself: Delegate work and keep talking Turn on client delegation, and use the session instructions to tell GPT-Live 1.1 when to ask for help. It asks by sending delegation-created. Answer with any text model on AI Gateway and return the result on the commentary channel for GPT-Live to speak. This example uses openai/gpt-6.2-sol and builds on the session above: Your application controls delegated work and its permissions, confirmations, and cancellation. Delegated model requests are billed separately through AI Gateway, and voice-session usage continues while they run. These snippets omit the WebSocket connection, event parsing, transcript assembly, audio streaming, and shutdown. See the GPT-Live guide for connection setup, audio streaming, and complete examples. Try GPT-Live 1.1 in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all models available on AI Gateway. Read more

## Hy4 Flash now available on AI Gateway for 50% off

DevFeed: [Hy4 Flash now available on AI Gateway for 50% off](<https://devfeed.tech/articles/hy4-flash-now-available-on-ai-gateway-for-50-off-136002.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/hy4-flash-now-available-on-ai-gateway-for-50-off>)

Author: Jerilyn Zheng

Published: 2026-11-16T17:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [bring-your-own-key](<https://devfeed.tech/tags/bring-your-own-key.md>), [byok](<https://devfeed.tech/tags/byok.md>), [caching](<https://devfeed.tech/tags/caching.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [cli](<https://devfeed.tech/tags/cli.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [discount](<https://devfeed.tech/tags/discount.md>), [flash](<https://devfeed.tech/tags/flash.md>), [hy4](<https://devfeed.tech/tags/hy4.md>), [tencent](<https://devfeed.tech/tags/tencent.md>)

### AI overview

Tencent's Hy4 Flash is available through AI Gateway at a stated introductory 50% discount on input, cached input, and output tokens through November 29, 2026. The lower-cost Hy4 model retains reasoning across turns, tool calling, structured outputs, prompt caching, image input, and a 1M-token context window; reasoning can be disabled for requests that do not need it. Use model ID `tencent/hy4-flash` in the AI SDK, or install the latest Vercel CLI, run `npx vercel ai-gateway setup`, and select the model in a supported coding agent. AI Gateway offers usage and cost tracking, retries, failover, routing, reporting, Zero Data Retention support, and API-key budgets. It says it applies no markup to provider pricing and charges no inference platform fee, including for BYOK requests.

### Source excerpt

Hy4 Flash from Tencent is now available on AI Gateway with an introductory 50% discount on input, cached input, and output tokens through November 29, 2026. Hy4 Flash is the new lower-cost model in the Hy4 family. It keeps the full Hy4 feature set at roughly a tenth of the price per token: reasoning with preserved thinking across turns, tool calling, structured outputs, prompt caching, and image input, with the same 1M-token context window. Reasoning can be turned off for requests that don't need it, which suits high-volume coding, agent, and document workloads. Use tencent/hy4-flash as the model name: To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select tencent/hy4-flash in the agent. See the coding agents guide for details. Try Hy4 Flash in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all language models available on AI Gateway. Read more

## Amazon Uncovers AI Agent Campaign Targeting LiteLLM, Dify, and Other Apps for API Keys

DevFeed: [Amazon Uncovers AI Agent Campaign Targeting LiteLLM, Dify, and Other Apps for API Keys](<https://devfeed.tech/articles/amazon-uncovers-ai-agent-campaign-targeting-litellm-dify-and-other-apps-for-api-keys-136007.md>)

Original publisher: [Read original article](<https://thehackernews.com/2026/11/amazon-uncovers-ai-agent-campaign.html>)

Author: The Hacker News

Published: 2026-11-16T07:34:21Z

Content type: news

Language: en

Sources: [Cyber Newsroom](<https://devfeed.tech/sources/cyber-newsroom.md>)

Topics: [token jacking](<https://devfeed.tech/topics/token-jacking.md>), [Secret Scanning](<https://devfeed.tech/topics/secret-scanning.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-apps](<https://devfeed.tech/tags/ai-apps.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [api-security](<https://devfeed.tech/tags/api-security.md>), [apps](<https://devfeed.tech/tags/apps.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [campaign](<https://devfeed.tech/tags/campaign.md>), [cloud-security](<https://devfeed.tech/tags/cloud-security.md>), [credential-theft](<https://devfeed.tech/tags/credential-theft.md>), [cve](<https://devfeed.tech/tags/cve.md>), [cyber-attacks](<https://devfeed.tech/tags/cyber-attacks.md>), [exposed](<https://devfeed.tech/tags/exposed.md>), [internet](<https://devfeed.tech/tags/internet.md>), [keys](<https://devfeed.tech/tags/keys.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [stored](<https://devfeed.tech/tags/stored.md>), [vulnerabilities](<https://devfeed.tech/tags/vulnerabilities.md>)

### AI overview

Amazon's MadPot decoys captured records showing AI agents targeting internet-exposed AI gateways and app builders--including LiteLLM, One API, Dify, Langflow and RAGFlow--to find provider keys, test them and pool working credentials. The agents' records describe techniques such as default credentials, unfinished installs, unauthenticated settings endpoints, known flaws, exposed files and SSRF to cloud metadata. Amazon could not verify the agents' reported outcomes; it assessed with moderate confidence that the operators were separate China-linked groups and that the agents used ARTEX. A separate Cloudflare investigation found ARTEX activity after an account-level CI/CD token was compromised; the token lacked IP restrictions and rate limits, but data returned by database dump requests and persistence success were not disclosed. Amazon advises changing defaults, patching software, scanning and rotating exposed secrets, and monitoring unusual usage or spending. Cloudflare recommends expiring API tokens and restricting client IPs; defenders can also check for named ARTEX Workers and published indicators. The excerpt also describes agent attempts to bypass crypto-exchange identity checks,

### Source excerpt

Amazon Uncovers AI Agent Campaign Targeting LiteLLM, Dify, and Other Apps for API Keys Attackers' AI agents have been hunting for weak spots in internet-exposed AI apps such as LiteLLM, One API, and Dify to steal the AI provider keys stored inside, according to the agents' own records. ... Read more Published Date: Nov 16, 2026 (11 hours, 18 minutes ago) Vulnerabilities has been mentioned in this article. CVE-2026-118383 CVE-2026-28797 CVE-2025-3248

## Hugging Face and OpenRouter join forces to make open-weight models easy to run

DevFeed: [Hugging Face and OpenRouter join forces to make open-weight models easy to run](<https://devfeed.tech/articles/hugging-face-and-openrouter-join-forces-to-make-open-weight-models-easy-to-run-137049.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/openrouter>)

Author: Célina Hanouti; Victor Mustar; Bertrand Chevrier; Lucain Pouget; Pierric Cistac; Eliott Coyac; Julien Chaumond

Published: 2026-11-16T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [inference-providers](<https://devfeed.tech/topics/inference-providers.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [apis](<https://devfeed.tech/tags/apis.md>), [artificial-analysis](<https://devfeed.tech/tags/artificial-analysis.md>), [artificial-analysis-intelligence-index](<https://devfeed.tech/tags/artificial-analysis-intelligence-index.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [audio](<https://devfeed.tech/tags/audio.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [hub](<https://devfeed.tech/tags/hub.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference-providers](<https://devfeed.tech/tags/inference-providers.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [join](<https://devfeed.tech/tags/join.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [openrouter](<https://devfeed.tech/tags/openrouter.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [science](<https://devfeed.tech/tags/science.md>)

### AI overview

Hugging Face now routes language-model requests through OpenRouter, which selects a provider using live price, speed, and reliability measurements and retries another provider when a call fails. Existing code and Hugging Face tokens continue to work; supported interfaces include OpenAI Chat Completions and Responses and Anthropic Messages. Users can select a routing policy or provider with a model-ID suffix, disable providers in settings, and use `X-HF-Session-Id` to keep a conversation on the provider holding its prompt cache. Pricing remains the provider's price with no Hugging Face markup, but users can no longer use their own provider API keys for language models. Image, video, and audio generation remain directly on Hugging Face Inference Providers. OpenRouter reports that its rebuilt routing cut capacity errors by 46% and improved time to first token by 16.9% and throughput by 16.3%. Open-weight models average about four times less per token in the cited figures, though the best open-weight model still scores below the best proprietary model on the cited benchmark, at substantially lower cost per task.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Best tools for routing and caching in AI coding workflows

DevFeed: [Best tools for routing and caching in AI coding workflows](<https://devfeed.tech/articles/best-tools-for-routing-and-caching-in-ai-coding-workflows-139353.md>)

Original publisher: [Read original article](<https://www.lyzr.ai/blog/best-tools-for-routing-and-caching-in-ai-coding-workflows/>)

Author: sujatha

Published: 2026-11-14T11:44:52Z

Content type: comparison

Language: en

Sources: [Lyzr.ai](<https://devfeed.tech/sources/lyzr-ai.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [Self Hosted AI Tools](<https://devfeed.tech/topics/self-hosted-ai-tools.md>), [generative-ai-tools](<https://devfeed.tech/topics/generative-ai-tools.md>), [cursor](<https://devfeed.tech/topics/cursor.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-workflows](<https://devfeed.tech/tags/ai-coding-workflows.md>), [best](<https://devfeed.tech/tags/best.md>), [caching](<https://devfeed.tech/tags/caching.md>), [coding](<https://devfeed.tech/tags/coding.md>), [routing](<https://devfeed.tech/tags/routing.md>), [tools](<https://devfeed.tech/tags/tools.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Routing sends each coding task to a model suited to its complexity; caching reduces repeated work by reusing answers or lowering the cost of resending context. The text recommends evaluating task-level rules, exact-match, semantic and provider prompt caching, cache invalidation, agent compatibility, and per-step cost and quality visibility. Tool tradeoffs differ: LiteLLM and Bifrost are self-hosted gateways; LiteLLM reserves several access controls for enterprise and the text notes 2026 security incidents, while Bifrost may skip caching for tool-call responses or conversations over three messages. Portkey offers configurable fallbacks and exact-match caching, but semantic caching has plan and request-size limits. OpenRouter provides hosted multi-provider failover without a proxy, though Claude Code is only guaranteed to work with Anthropic's first-party provider. Lyzr combines workflow-level model assignment and budgets with OpenController caching and failover, plus self-hosted or air-gapped deployment options. The text cautions that semantic cache matches can return incorrect coding answers and advises limiting them to low-risk repetitive requests.

### Source excerpt

AI coding spend has stopped being a rounding error. Jellyfish's H1 2026 report found that an AI coding user's monthly bill rose from about $5...

## Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

DevFeed: [Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard](<https://devfeed.tech/articles/route-coding-agent-requests-to-the-right-model-on-amazon-bedrock-with-nvidia-nemo-switchyard-133426.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/route-coding-agent-requests-to-the-right-model-on-amazon-bedrock-with-nvidia-nemo-switchyard/>)

Author: Aswath Ram A Srinivasan

Published: 2026-11-13T22:30:12Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [NeMo](<https://devfeed.tech/topics/nemo.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Nemotron](<https://devfeed.tech/topics/nemotron.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cdk](<https://devfeed.tech/tags/aws-cdk.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cost](<https://devfeed.tech/tags/cost.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [nemo](<https://devfeed.tech/tags/nemo.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [route](<https://devfeed.tech/tags/route.md>), [routing](<https://devfeed.tech/tags/routing.md>), [solved](<https://devfeed.tech/tags/solved.md>), [switchyard](<https://devfeed.tech/tags/switchyard.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

The article's main takeaway is to judge coding-agent routing by cost per solved task, since multi-turn history, retries, and failed runs make token price alone misleading. It walks through deploying LiteLLM with NVIDIA NeMo Switchyard on Amazon Bedrock using AWS CDK, then comparing a routed model group with single-model baselines. Stage routing uses recent tool results to choose between exactly two candidates; it starts with the efficient model when signals are inconclusive and can escalate after failures. Random routing splits requests among candidates by weight. The sample's benchmark estimates cost from token usage and a price table, and counts a task solved when its tests pass within the turn limit. Results are illustrative: repeat runs, inspect hard tasks and truncation, and treat costs as estimates rather than billing data. The Switchyard integration is experimental, supports only specified message shapes and Chat Completions paths, and is presented for non-production use.

### Source excerpt

Coding agents work in tasks, and the number that matters is cost per solved task, not price per token. In this post, you use the AWS CDK to deploy LiteLLM with the NVIDIA NeMo Switchyard routing plugin on Amazon Bedrock, then measure cost per solved task for a routed model group against single-model baselines.

## How Much Latency Does a Self-Hosted LLM Gateway Add? Langfuse vs LiteLLM vs AcruxCore

DevFeed: [How Much Latency Does a Self-Hosted LLM Gateway Add? Langfuse vs LiteLLM vs AcruxCore](<https://devfeed.tech/articles/how-much-latency-does-a-self-hosted-llm-gateway-add-langfuse-vs-litellm-vs-acruxcore-134431.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/how-much-latency-does-a-self-hosted-llm-gateway-add-langfuse-vs-litellm-vs-acruxcore/>)

Author: Talha Anwar

Published: 2026-11-13T15:40:26Z

Content type: news

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [gateway](<https://devfeed.tech/topics/gateway.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Self-hosted](<https://devfeed.tech/topics/self-hosted.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [budget](<https://devfeed.tech/tags/budget.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [langfuse](<https://devfeed.tech/tags/langfuse.md>), [latency](<https://devfeed.tech/tags/latency.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-gateway](<https://devfeed.tech/tags/llm-gateway.md>), [performance-scaling](<https://devfeed.tech/tags/performance-scaling.md>), [self-hosted](<https://devfeed.tech/tags/self-hosted.md>)

### AI overview

In a controlled test against the same OpenAI model, LiteLLM added 61-67 ms to non-streaming calls, AcruxCore 15-24 ms, and Langfuse's measured cost could not be distinguished from zero. For first streamed tokens, no gateway showed a consistent added delay across all runs. The result depended on controlling network effects: one OpenAI address added about 250 ms on new connections, so the test pinned the address and kept the direct client's connection open to match gateway behavior. It compared paired calls within each round and used bootstrap intervals; intervals including zero indicate uncertainty, not proof of no cost. Logging levels showed no clear effect, and overlapping intervals prevent ranking Langfuse against AcruxCore. Findings are limited to short calls, one network and latency; the test did not measure performance under load.

### Source excerpt

null Continue reading How Much Latency Does a Self-Hosted LLM Gateway Add? Langfuse vs LiteLLM vs AcruxCore on SitePoint.

## Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

DevFeed: [Building a self-service LLM gateway with Amazon Bedrock and LiteLLM](<https://devfeed.tech/articles/building-a-self-service-llm-gateway-with-amazon-bedrock-and-litellm-130770.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/building-a-self-service-llm-gateway-with-amazon-bedrock-and-litellm/>)

Author: Sitaraman Vijay Krishna

Published: 2026-11-12T18:41:12Z

Content type: tutorial

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [self-service](<https://devfeed.tech/topics/self-service.md>), [internal developer portal](<https://devfeed.tech/topics/internal-developer-portal.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [Amazon RDS](<https://devfeed.tech/topics/amazon-rds.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-fargate](<https://devfeed.tech/tags/aws-fargate.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cost](<https://devfeed.tech/tags/cost.md>), [developers](<https://devfeed.tech/tags/developers.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-gateway](<https://devfeed.tech/tags/llm-gateway.md>), [self-service](<https://devfeed.tech/tags/self-service.md>), [service](<https://devfeed.tech/tags/service.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a self-service LLM gateway pattern that puts Amazon Bedrock behind LiteLLM's OpenAI-compatible API on AWS Fargate. Developers use a single base URL and per-user keys; LiteLLM manages key budgets and rate limits, PostgreSQL stores key and spend state, Valkey provides caching and distributed rate limiting, and Bedrock inference profiles provide model-level quota isolation and routing. Langfuse captures prompt-level traces while CloudWatch tracks operational metrics; prompt logging requires suitable redaction, retention, and access controls. Key implementation requirements include LiteLLM Enterprise for SSO-based self-service key issuance, explicit per-model token pricing for budget enforcement, PostgreSQL for persistent state, and IAM permissions for both inference-profile and underlying model resources. The authors recommend long request timeouts for extended thinking, retrying throttled requests on the same model, and separating automation traffic. The post offers illustrative snippets rather than a complete deployable stack, and its examples use Claude models.

### Source excerpt

Build a self-service LLM gateway with LiteLLM on AWS Fargate that puts Amazon Bedrock foundation models behind an OpenAI-compatible API. Developers get instant access through a single API key, while platform teams gain per-user cost controls, quota isolation, and observability.

## Breaking the Semantic Cache: A LiteLLM Case Study (CVE-2026-89032)

DevFeed: [Breaking the Semantic Cache: A LiteLLM Case Study (CVE-2026-89032)](<https://devfeed.tech/articles/breaking-the-semantic-cache-a-litellm-case-study-cve-2026-89032-130461.md>)

Original publisher: [Read original article](<https://blog.nviso.eu/2026/11/12/breaking-the-semantic-cache-a-litellm-case-study-cve-2026-89032/>)

Author: Tanguy Snoeck

Published: 2026-11-12T12:16:18Z

Content type: article

Language: en

Sources: [Cyber Newsroom](<https://devfeed.tech/sources/cyber-newsroom.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [application-security](<https://devfeed.tech/tags/application-security.md>), [appsec](<https://devfeed.tech/tags/appsec.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [attack](<https://devfeed.tech/tags/attack.md>), [bank](<https://devfeed.tech/tags/bank.md>), [bash](<https://devfeed.tech/tags/bash.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cve](<https://devfeed.tech/tags/cve.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-cache-poisoning](<https://devfeed.tech/tags/semantic-cache-poisoning.md>), [vulnerabilities](<https://devfeed.tech/tags/vulnerabilities.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

Semantic caches can leak data or spread attacker-controlled outputs when semantically similar requests share cache entries. The article explains that a low similarity threshold increases hits but can match questions that should be treated differently. It reports CVE-2026-89032 as breaking LiteLLM's API-key cache isolation; the demonstrated attacks on vulnerable v1.99.0 retrieve a victim's PII, spread a fabricated answer, or replay a poisoned tool call. The examples use threshold 0.2 for simplicity, while the article says typical deployments use higher values, making attacks harder. It identifies LiteLLM v1.101.0-rc.1 as containing a patch and recommends avoiding semantic caching where feasible; otherwise isolate cache buckets per user, tune the threshold, filter sensitive responses, and never cache tool calls. The coding-agent example depends on specific routes and pi v0.45.3; the article says other agents are not necessarily affected.

### Source excerpt

Breaking the Semantic Cache: A LiteLLM Case Study (CVE-2026-89032) Introduction Issues related to web cache misconfiguration have been a prolific ground for vulnerabilities for the last decade and generated entire new attack classes as a result. Historically, web cac ... Read more Published Date: Nov 12, 2026 (3 hours, 3 minutes ago) Vulnerabilities has been mentioned in this article. CVE-2026-89032

## Inkling 2 Small now available on AI Gateway

DevFeed: [Inkling 2 Small now available on AI Gateway](<https://devfeed.tech/articles/inkling-2-small-now-available-on-ai-gateway-131130.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/inkling-2-small-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-12T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [bring-your-own-key](<https://devfeed.tech/tags/bring-your-own-key.md>), [charts](<https://devfeed.tech/tags/charts.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [coding-tasks](<https://devfeed.tech/tags/coding-tasks.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [development](<https://devfeed.tech/tags/development.md>), [image](<https://devfeed.tech/tags/image.md>), [inkling-2-small](<https://devfeed.tech/tags/inkling-2-small.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-model](<https://devfeed.tech/tags/reasoning-model.md>), [screenshots](<https://devfeed.tech/tags/screenshots.md>)

### AI overview

Thinking Machines' Inkling 2 Small is available through AI Gateway as `thinkingmachines/inkling-2-small`. It accepts text and images and returns text, supporting extended coding tasks, tool-assisted research, and deliverables such as documents and spreadsheets. Its visual capabilities include reading image text, interpreting charts and infographics, inspecting interfaces, and using screenshots to guide code revisions. For Claude Code, Codex, Cursor, and other agents, the excerpt says to install the latest Vercel CLI, run `npx vercel ai-gateway setup`, then select the model. AI Gateway offers usage and cost tracking, retries, failover, routing rules, reporting, Zero Data Retention support, and API-key budgets. It says pricing matches providers with no markup or platform fee, including BYOK inference.

### Source excerpt

Inkling 2 Small from Thinking Machines is now available on AI Gateway. The multimodal reasoning model supports coding agents, visual understanding, and professional knowledge work, with text and image inputs and text output. Inkling 2 Small can work through extended coding tasks, use tools to gather information, and turn that information into documents, spreadsheets, and other deliverables. Its visual capabilities include reading text in images, interpreting charts and infographics, and inspecting interfaces. For visual development, it can use screenshots to check the output of the code it writes and guide the next revision. Use thinkingmachines/inkling-2-small as the model name: To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select thinkingmachines/inkling-2-small in the agent. See the coding agents guide for details. Try Inkling 2 Small in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all language models available on AI Gateway. Read more

## Does paying more for a coding model deliver better performance? | Blog | Fivetran

DevFeed: [Does paying more for a coding model deliver better performance? | Blog | Fivetran](<https://devfeed.tech/articles/does-paying-more-for-a-coding-model-deliver-better-performance-blog-fivetran-136769.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/does-paying-more-for-a-coding-model-deliver-better-performance>)

Author: Aleksandar Kaplar

Published: 2026-11-11T16:05:32Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [claude-code](<https://devfeed.tech/tags/claude-code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [openai-models](<https://devfeed.tech/tags/openai-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [study](<https://devfeed.tech/tags/study.md>), [value](<https://devfeed.tech/tags/value.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

In Fivetran's blinded two-week study, higher-priced OpenAI coding models showed only a modest, inconclusive preference edge in engineers' everyday work. Thirty-nine engineers were assigned GPT 5.6 Luna, Terra, or Sol through Claude Code; 27 surveyed after testing both assigned models. Sol won both matchups, but nearly half the preferences were weak, and the small vote counts did not establish a clear ranking. Typical weekly spend rose from about $5 on Luna to $41 on Terra and $168 on Sol; PR counts did not establish productivity differences. Feedback pointed to tradeoffs: deeper planning helped large refactors, while responsiveness and scope control mattered for focused tasks, where heavier models could overreach. The proxy setup also caused friction with non-Anthropic models, and each model handled the full agent loop at its own rates. The study did not measure potential savings from mixed-model workflows; Fivetran proposes a follow-up that holds execution models, tools, and budgets constant.

### Source excerpt

In a blinded workplace study, we tested 3 tiers of OpenAI models and learned that price is a poor proxy for value.

## What Breaks When a Self-Hosted LLM Gateway Hits 1,000 Concurrent Streams

DevFeed: [What Breaks When a Self-Hosted LLM Gateway Hits 1,000 Concurrent Streams](<https://devfeed.tech/articles/what-breaks-when-a-self-hosted-llm-gateway-hits-1-000-concurrent-streams-126634.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/what-breaks-when-a-self-hosted-llm-gateway-hits-1-000-concurrent-streams/>)

Author: Talha Anwar

Published: 2026-11-10T11:31:05Z

Content type: article

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Self-hosted](<https://devfeed.tech/topics/self-hosted.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [1-7](<https://devfeed.tech/tags/1-7.md>), [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [403](<https://devfeed.tech/tags/403.md>), [about](<https://devfeed.tech/tags/about.md>), [act](<https://devfeed.tech/tags/act.md>), [after](<https://devfeed.tech/tags/after.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [amd](<https://devfeed.tech/tags/amd.md>), [amd-ryzen](<https://devfeed.tech/tags/amd-ryzen.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [developer-tools](<https://devfeed.tech/tags/developer-tools.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-gateway](<https://devfeed.tech/tags/llm-gateway.md>), [self-hosted](<https://devfeed.tech/tags/self-hosted.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

In a 1,000-stream test with each gateway capped at 2 CPUs, Langfuse completed 261 calls and LiteLLM 403 as shipped; both hit Linux's 1,024 open-file soft limit, which their processes did not raise at startup. Setting a 65,536 nofile limit let Langfuse complete 1,000 streams. LiteLLM also failed during cold-cache bursts: its 10-connection-per-worker Postgres pool queued key lookups until some exceeded the 10-second deadline and returned 503s. Warming each virtual key eliminated failures in the tested 500-stream run; changing cache TTL, lookup deadline, or lookup concurrency are possible settings to investigate, but were not tested as fixes. After limits were raised, CPU and stack-wide resource use remained important: first-token latency rose under load, and supporting services added CPU consumption.

### Source excerpt

null Continue reading What Breaks When a Self-Hosted LLM Gateway Hits 1,000 Concurrent Streams on SitePoint.

## Gemini 4 Carbon now available on AI Gateway

DevFeed: [Gemini 4 Carbon now available on AI Gateway](<https://devfeed.tech/articles/gemini-4-carbon-now-available-on-ai-gateway-126761.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/gemini-4-carbon-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-10T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [cli](<https://devfeed.tech/tags/cli.md>), [code](<https://devfeed.tech/tags/code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [failover](<https://devfeed.tech/tags/failover.md>), [full-stack](<https://devfeed.tech/tags/full-stack.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gemini-4-carbon](<https://devfeed.tech/tags/gemini-4-carbon.md>), [google](<https://devfeed.tech/tags/google.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

Google's Gemini 4 Carbon is available through Vercel AI Gateway under the model name `google/gemini-4-carbon`, with 50% off through November 30, 2026. The text describes a 1M-token context window and capabilities for multi-stage software engineering and cybersecurity work, including finding vulnerabilities, assessing risk, and developing patches. Google reports more reliable multi-step planning, fewer unsupported claims, and better handling of ambiguous requirements; these are reported claims, not independently established results. The text shows use through the AI SDK and says Claude Code, Codex, Cursor, and other agents can be configured by installing the latest Vercel CLI, running `npx vercel ai-gateway setup`, and selecting the model. AI Gateway offers usage and cost tracking, retries, failover, performance optimizations, reporting, Zero Data Retention support, API key budgets, and routing rules. It says pricing follows providers without markup or an inference platform fee, including BYOK requests.

### Source excerpt

Gemini 4 Carbon from Google is now available on AI Gateway. It is available at 50% off through November 30, 2026: Gemini 4 Carbon is built for complex software engineering, sustained agentic work, and cybersecurity defense. It has a 1M-token context window for working with large codebases and long inputs. On engineering tasks, Carbon carries demanding work through multiple stages, with improvements in coding, full-stack web development, and scientific reasoning. Its cybersecurity capabilities span finding vulnerabilities, validating their risk, and developing patches. Google also reports more reliable execution of multi-step plans, fewer unsupported claims, and better collaboration when requirements are ambiguous, helping agents follow through while respecting project constraints. Use google/gemini-4-carbon as the model name: To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select google/gemini-4-carbon in the agent. See the coding agents guide for details. Try Gemini 4 Carbon in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all language models available on AI Gateway. Read more

## Does paying more for a coding model deliver better performance? | Blog | Fivetran

DevFeed: [Does paying more for a coding model deliver better performance? | Blog | Fivetran](<https://devfeed.tech/articles/does-paying-more-for-a-coding-model-deliver-better-performance-blog-fivetran-123779.md>)

Original publisher: [Read original article](<https://www.fivetran.com/blog/does-paying-more-for-a-coding-model-deliver-better-performance>)

Author: Aleksandar Kaplar

Published: 2026-11-09T16:02:24Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blog](<https://devfeed.tech/tags/blog.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [claude-sonnet](<https://devfeed.tech/tags/claude-sonnet.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [openai](<https://devfeed.tech/tags/openai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [study](<https://devfeed.tech/tags/study.md>), [value](<https://devfeed.tech/tags/value.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

In Fivetran's two-week blinded workplace study, a higher model price did not translate into a decisive day-to-day preference advantage. Thirty-nine engineers were assigned GPT 5.6 Luna, Terra, or Sol in Claude Code; 27 who used both assigned models completed the preference survey. Sol won both matchups, but nearly half of preferences were weak and the small vote counts did not establish a clear ranking. Typical weekly spend rose from about $5 on Luna to $41 on Terra and $168 on Sol. PR counts varied, but the study says they cannot establish productivity rankings because they depend on the work undertaken. Feedback pointed to tradeoffs: deeper planning helped large refactors, while speed and scope control mattered on focused tasks, where heavier models sometimes overreached. The proxy setup also introduced friction, and each model handled routine agent work at its own rates; the study did not measure potential savings from mixed-model routing. Its results apply to this setup and do not show that the models are equally capable.

### Source excerpt

In a blinded workplace study, we tested 3 tiers of OpenAI models and learned that price is a poor proxy for value.

## Black October 2026: How AI sped up bug hunting and attacks

DevFeed: [Black October 2026: How AI sped up bug hunting and attacks](<https://devfeed.tech/articles/black-october-2026-how-ai-sped-up-bug-hunting-and-attacks-123472.md>)

Original publisher: [Read original article](<https://workos.com/blog/black-october-2026-ai-security>)

Author: WorkOS

Published: 2026-11-09T00:00:00Z

Content type: news

Language: en

Sources: [WorkOS](<https://devfeed.tech/sources/workos-blog.md>)

Topics: [Cybersecurity](<https://devfeed.tech/topics/cybersecurity.md>), [ai-assisted attacks](<https://devfeed.tech/topics/ai-assisted-attacks.md>), [litellm attack](<https://devfeed.tech/topics/litellm-attack.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [Threat Hunting & Intel](<https://devfeed.tech/topics/threat-hunting-intel.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [black](<https://devfeed.tech/tags/black.md>), [black-october](<https://devfeed.tech/tags/black-october.md>), [bug](<https://devfeed.tech/tags/bug.md>), [bug-hunting](<https://devfeed.tech/tags/bug-hunting.md>), [ca](<https://devfeed.tech/tags/ca.md>), [critical](<https://devfeed.tech/tags/critical.md>), [gitlab](<https://devfeed.tech/tags/gitlab.md>), [here-s](<https://devfeed.tech/tags/here-s.md>), [hit](<https://devfeed.tech/tags/hit.md>), [keys](<https://devfeed.tech/tags/keys.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [node](<https://devfeed.tech/tags/node.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [october](<https://devfeed.tech/tags/october.md>), [october-2026](<https://devfeed.tech/tags/october-2026.md>), [up](<https://devfeed.tech/tags/up.md>)

### AI overview

Black October's flaws came from familiar causes, while AI appears to have accelerated discovery and intrusions. PHP's October 21 releases fix two critical unauthenticated remote code execution bugs; the advisory says all web SAPIs are affected and updating is the only complete mitigation. Node.js patched 10 vulnerabilities, including a Brotli flaw later listed by CISA as actively exploited. GitLab patched a critical unauthenticated code execution issue; self-compiled installations also need updated Ruby. Chrome and Mozilla blocked affected eMudhra CA keys, and eMudhra said it would exit global SSL/TLS certificates. LiteLLM reported that a malicious package and Docker tags were briefly available; its account describes stolen developer tokens, exposed logs, and a release workflow compromise. Wiz says AI helped find and validate PHP exploits, though its claim that one researcher compressed months of work into a week lacks published metrics. ARTEX reports describe machine-speed intrusions, with Google noting a tradeoff between speed and stealth. The text says two of the five core disclosures had no reported AI involvement.

### Source excerpt

Critical flaws in PHP, Node.js and GitLab, a poisoned LiteLLM release and weak CA keys all hit in October 2026. Here's what AI had to do with it.

## DeepSeek V4.1 Flash 1017 Ultrafast now available on AI Gateway

DevFeed: [DeepSeek V4.1 Flash 1017 Ultrafast now available on AI Gateway](<https://devfeed.tech/articles/deepseek-v4-1-flash-1017-ultrafast-now-available-on-ai-gateway-122211.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/deepseek-v4-1-flash-1017-ultrafast-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-07T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [accelerated](<https://devfeed.tech/tags/accelerated.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [code](<https://devfeed.tech/tags/code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [data-retention](<https://devfeed.tech/tags/data-retention.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v4-1-flash](<https://devfeed.tech/tags/deepseek-v4-1-flash.md>), [deepseek-v4-1-flash-1017](<https://devfeed.tech/tags/deepseek-v4-1-flash-1017.md>), [failover](<https://devfeed.tech/tags/failover.md>)

### AI overview

DeepSeek V4.1 Flash 1017 Ultrafast is available in public preview on AI Gateway through Fireworks' accelerated serverless endpoint, with no waitroom or session time limit. The excerpt says it can generate upwards of 1,000 tokens per second and has a 1M-token context window, making it suited to interactive coding, reasoning, and agentic workflows that need to advance quickly. Use `deepseek/deepseek-v4.1-flash-1017-ultrafast` as the model name; for Claude Code, Codex, Cursor, and other agents, install the latest Vercel CLI, run `npx vercel ai-gateway setup`, then select that model. AI Gateway offers a unified API, usage and cost tracking, retries, failover, routing, and reporting. It reflects provider pricing without a markup or inference platform fee, including for BYOK requests.

### Source excerpt

DeepSeek V4.1 Flash 1017 Ultrafast is now available on AI Gateway through Fireworks. This model can generate upwards of 1,000 tokens per second. Ultrafast serves the October 17 update of DeepSeek V4.1 Flash through Fireworks' accelerated serverless endpoint. It is suited to interactive coding, reasoning, and agentic workflows where applications need to generate output and move to the next step quickly. The model has a 1M-token context window for working with large codebases and long inputs. It is available in public preview without a waitroom or session time limit. Use it by setting deepseek/deepseek-v4.1-flash-1017-ultrafast as the model name: To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select deepseek/deepseek-v4.1-flash-1017-ultrafast in the agent. See the coding agents guide for details. Try DeepSeek V4.1 Flash 1017 Ultrafast in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all language models available on AI Gateway. Read more

## Ampersand from Humans& now available on AI Gateway

DevFeed: [Ampersand from Humans& now available on AI Gateway](<https://devfeed.tech/articles/ampersand-from-humans-now-available-on-ai-gateway-121733.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/humans-and-ampersand-now-available-on-ai-gateway>)

Author: Jerilyn Zheng

Published: 2026-11-06T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [codex](<https://devfeed.tech/topics/codex.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [ampersand](<https://devfeed.tech/tags/ampersand.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [application](<https://devfeed.tech/tags/application.md>), [chat](<https://devfeed.tech/tags/chat.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [cli](<https://devfeed.tech/tags/cli.md>), [code](<https://devfeed.tech/tags/code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [conversations](<https://devfeed.tech/tags/conversations.md>)

### AI overview

Ampersand is a group-conversation model that chooses on each turn whether to contribute or wait. Applications expose caller-defined `send_message` and `pass` tools, execute the chosen action, and record its result. Keep tool choice at the default: Ampersand accepts `auto` and `none`, but rejects `required` and named choices. The excerpt shows integration through the AI SDK, OpenAI-compatible Chat Completions API, and coding agents using `humansand/ampersand-v0.1`. For deployed chat, it recommends durable storage for responses, history, and delivery receipts; sequential processing per conversation; retrying uncertain delivery with the same receipt; and checking access before posting. AI Gateway offers usage and cost tracking, retries, failover, performance options, reporting, budgets, and routing rules. The excerpt says there is no inference markup or platform fee, including for BYOK, and that Ampersand is free through November 13, 2026.

### Source excerpt

Ampersand from Humans& is now available on AI Gateway for free through November 13, 2026. Ampersand is a language model built to take part in group conversations, such as a team chat, and to decide on each turn whether to speak or stay quiet. It is designed to understand individual and shared goals, help people work through disagreements, and recognize when a contribution would be useful. That can mean asking a clarifying question, sharing relevant context, or waiting while people respond to one another. Your application exposes these actions as caller-defined function tools. send_message delivers a contribution, while pass leaves the conversation quiet until the next human message. Your application executes both tools and records their results. Leave tool choice at its default, since Ampersand accepts auto and none but rejects required and named tool choices. Call it through the AI SDK, the OpenAI-compatible Chat Completions API, or a coding agent. Use humansand/ampersand-v0.1 as the model name: The first human message invites a contribution; the second asks Ampersand to wait. In a chat application, call the handler when a new human message arrives. Handling pass records the waiting action without posting a message or starting another model turn. For a deployed chat application, persist the response, history, and delivery receipts in durable storage. Retry uncertain delivery with the saved action and the same receipt, process messages sequentially per conversation, and check conversation access before posting. To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select humansand/ampersand-v0.1 in the agent. See the coding agents guide for details. Try Ampersand in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, budgets for API keys

## Token Pass credit now applies automatically across AI Gateway

DevFeed: [Token Pass credit now applies automatically across AI Gateway](<https://devfeed.tech/articles/token-pass-credit-now-applies-automatically-across-ai-gateway-119609.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/token-pass-credit-now-applies-automatically-across-ai-gateway>)

Author: Rohan Taneja

Published: 2026-11-05T22:34:49.121455Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [litellm](<https://devfeed.tech/topics/litellm.md>), [API keys](<https://devfeed.tech/topics/api-keys.md>), [API Monetization](<https://devfeed.tech/topics/api-monetization.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [billing](<https://devfeed.tech/tags/billing.md>), [can](<https://devfeed.tech/tags/can.md>), [cli](<https://devfeed.tech/tags/cli.md>), [created](<https://devfeed.tech/tags/created.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [data](<https://devfeed.tech/tags/data.md>), [data-retention](<https://devfeed.tech/tags/data-retention.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [fall](<https://devfeed.tech/tags/fall.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [featured](<https://devfeed.tech/tags/featured.md>), [follow](<https://devfeed.tech/tags/follow.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [pass](<https://devfeed.tech/tags/pass.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

Token Pass credit now applies automatically to included AI Gateway model usage authenticated with an API key, OIDC token, or fx login; a dedicated Token Pass key is optional. After the billing-period credit runs out, standard API keys and OIDC tokens fall back to team AI Gateway credits, while Token Pass keys stop until the next period. Token Pass-served requests retain ZDR. Fallback requests follow team settings and, with team-wide ZDR enabled, route only to ZDR providers. The feature is available to Token Pass subscribers on Pro teams; users who set up a Token Pass key must switch to a standard key or OIDC token to receive fallback.

### Source excerpt

Token Pass credit now applies automatically to included model usage through AI Gateway, whether a request authenticates with an API key, OIDC token, or fx login. No dedicated Token Pass API key is required, though you can still create one from the dashboard or the CLI for keys scoped only to featured models and providers. When Token Pass credit runs out for the billing period, requests from a standard key or OIDC token fall back to your team's AI Gateway credits. Token Pass keys still stop until the next billing period. Requests served through the Token Pass provider remain Zero Data Retention (ZDR). Fallback requests follow your team's setting. With team-wide ZDR enabled, they route only to ZDR providers. Available now for Token Pass subscribers on Pro teams. If you created a Token Pass key during setup, switch to a standard API key or OIDC token to get fallback. Learn more in the Token Pass documentation. Read more

## Kimi K3.1 Pro now has more ZDR providers on AI Gateway

DevFeed: [Kimi K3.1 Pro now has more ZDR providers on AI Gateway](<https://devfeed.tech/articles/kimi-k3-1-pro-now-has-more-zdr-providers-on-ai-gateway-117141.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/kimi-k3-1-pro-now-has-more-zdr-providers-on-ai-gateway>)

Author: Jerilyn Zheng

Published: 2026-11-04T17:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [Chat completions endpoints](<https://devfeed.tech/topics/chat-completions-endpoints.md>), [vercel ai sdk](<https://devfeed.tech/topics/vercel-ai-sdk.md>)

Tags: [3d-modeling](<https://devfeed.tech/tags/3d-modeling.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [failover](<https://devfeed.tech/tags/failover.md>), [financial](<https://devfeed.tech/tags/financial.md>), [image](<https://devfeed.tech/tags/image.md>), [input](<https://devfeed.tech/tags/input.md>), [install](<https://devfeed.tech/tags/install.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [k3](<https://devfeed.tech/tags/k3.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [kimi-k3-1-pro](<https://devfeed.tech/tags/kimi-k3-1-pro.md>), [providers](<https://devfeed.tech/tags/providers.md>)

### AI overview

Kimi K3.1 Pro and K3.1 Pro Highspeed are available through AI Gateway from additional ZDR providers; Moonshot AI had previously been the only provider serving them. Both models accept image input and have a 1M-token context window. The text highlights coding, 3D modeling, and financial analysis, and says Highspeed is suited to interactive work where response time matters. They can be called through the AI SDK, Chat Completions, Responses, Anthropic Messages, or connected coding agents. ZDR can be required per request with the gateway provider option or for all team requests through dashboard settings. When enabled, routing is limited to providers with a ZDR agreement, and requests fail if none is available for the model.

### Source excerpt

Kimi K3.1 Pro and its higher-speed version, Kimi K3.1 Pro Highspeed, are now available on AI Gateway from more Zero Data Retention (ZDR) providers. Until today, Moonshot AI was the only provider serving either model. K3.1 Pro improves on earlier Kimi models in coding, 3D modeling, and financial analysis, and can carry long agent runs from a small fix to a full project. Both models have a 1M-token context window and accept image input, and Highspeed suits interactive work like coding agents, where response time matters. Call either model through the AI SDK, the Chat Completions, Responses, and Anthropic Messages APIs, or a coding agent connected to AI Gateway. For Highspeed, use moonshotai/kimi-k3.1-pro-highspeed in place of the model name below. Use moonshotai/kimi-k3.1-pro as the model name: Setting zeroDataRetention in the gateway provider options requires ZDR for that request. To require it for every request your team sends, including requests from coding agents, turn on ZDR in the AI Gateway dashboard settings. With either one on, AI Gateway routes only to providers with a ZDR agreement. If none is available for the model, the request fails instead of going to a provider without one. To use it in Claude Code, Codex, Cursor, and more, install the latest Vercel CLI and run setup: Then select moonshotai/kimi-k3.1-pro or moonshotai/kimi-k3.1-pro-highspeed in the agent. See the coding agents guide for details. Try Kimi K3.1 Pro in the model playground. AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, Zero Data Retention support, budgets for API keys, routing rules, and more. AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests. You can view all language models available on AI Gateway. Read more

## Are Open LLMs Still Underrated, or Just Not Ready for Enterprise?

DevFeed: [Are Open LLMs Still Underrated, or Just Not Ready for Enterprise?](<https://devfeed.tech/articles/are-open-llms-still-underrated-or-just-not-ready-for-enterprise-118017.md>)

Original publisher: [Read original article](<https://medium.com/google-developer-experts/are-open-llms-still-underrated-or-just-not-ready-for-enterprise-1702a5651d54?source=rss----a67bd6fa7d58---4>)

Author: Geeta Kakrani

Published: 2026-11-04T14:26:59Z

Content type: article

Language: en

Sources: [Google Developer Experts](<https://devfeed.tech/sources/google-developer-experts-medium.md>)

Topics: [llamafile](<https://devfeed.tech/topics/llamafile.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [api](<https://devfeed.tech/tags/api.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [article](<https://devfeed.tech/tags/article.md>), [attack](<https://devfeed.tech/tags/attack.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [best](<https://devfeed.tech/tags/best.md>), [bill](<https://devfeed.tech/tags/bill.md>), [budget](<https://devfeed.tech/tags/budget.md>), [but](<https://devfeed.tech/tags/but.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [cases](<https://devfeed.tech/tags/cases.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [google](<https://devfeed.tech/tags/google.md>), [google-gemini](<https://devfeed.tech/tags/google-gemini.md>), [google-gemma](<https://devfeed.tech/tags/google-gemma.md>), [just](<https://devfeed.tech/tags/just.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [open](<https://devfeed.tech/tags/open.md>)

### AI overview

Open LLMs are suited to routine, high-volume work as one layer in a routed AI system, but the evidence does not support replacing frontier models across the board. The article recommends handling repeat requests with caches or rules, routing routine tasks to small open models, reserving frontier models for hard reasoning, and setting token budgets. Reported routing research shows large potential cost reductions, but savings depend on workload and hosting utilization. Self-hosting shifts costs to compute, engineering, quality checks, latency tuning, and security review; cited break-even and performance figures are third-party estimates or benchmark results, not universal comparisons. The article advises testing models in shadow mode on an organization's own tasks, measuring cost per task and utilization, and starting with a narrow pilot. Its survey is small and mostly North American, and its energy-savings evidence is described as thin and largely modeled.

### Source excerpt

Geeta Kakrani Open models already run in production at 42% of enterprises, yet they take only about 5% of what those enterprises spend on AI. That gap, not model quality, is the real story. In Futurum's September 2026 survey, production users said open weights now carry about 34% of their AI tokens, up from 23% a year earlier. Google's Gemma runs in 55% of production accounts, up from 35%. But the money did not follow: the tokens move to open weights, and the budget reappears as servers, compute and engineering time. The survey is small (200 enterprises, 83 production users, mostly North America), so read it as a signal, not a verdict. So are open LLMs underrated, or just not ready? The short answer is neither. They are ready for the routine layer of an AI system, and they pay off only when they sit behind a router with a budget on top. This article covers what open models really cost (hosting, quality, latency, skills), and how to recover from each cost. The promise: stop paying frontier prices for routine work Most agent traffic is routine: tagging, extraction, formatting, simple lookups. Sending all of it to the biggest model is the main reason API bills climb. Open models attack exactly this part, and the research on routing supports the idea. FrugalGPT reports matching the best single LLM (for example GPT-4) with up to 98% lower cost, using prompt adaptation, model approximation and a cascade of models. RouteLLM reports cutting cost by more than 2x in some cases without compromising response quality. In the Futurum survey, low-complexity workloads grew from 39% to 51% of production users' open-weight usage, so enterprises are already using open models where it is safest. These are best-case, benchmark-style results. Your savings depend on how much of your traffic is truly routine, and on what the open model costs to run, which is the next section. The hidden bill: four costs the API price hides An open model has no per-token fee, but it is not free. The cost mo

## 6 Best Platforms for LLM Routing and Load Balancing

DevFeed: [6 Best Platforms for LLM Routing and Load Balancing](<https://devfeed.tech/articles/6-best-platforms-for-llm-routing-and-load-balancing-116536.md>)

Original publisher: [Read original article](<https://www.lyzr.ai/blog/best-platforms-for-llm-routing-and-load-balancing/>)

Author: Lyzr Team

Published: 2026-11-04T11:27:45Z

Content type: comparison

Language: en

Sources: [Lyzr.ai](<https://devfeed.tech/sources/lyzr-ai.md>)

Topics: [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [air-gapped](<https://devfeed.tech/tags/air-gapped.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [best](<https://devfeed.tech/tags/best.md>), [costs](<https://devfeed.tech/tags/costs.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [openrouter](<https://devfeed.tech/tags/openrouter.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [routing](<https://devfeed.tech/tags/routing.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

The six platforms cover distinct routing layers. Lyzr OpenController combines model selection, answer validation with escalation, fallback, budgets and local deployment; it can route a task to a small model first and send failures to a frontier model. Kong AI Gateway offers weighted, latency, usage, sticky-session, semantic and priority-group balancing. Bifrost's enterprise tier recalculates provider and key weights every five seconds using health signals. LiteLLM Router has several routing strategies and cross-region retries, but the excerpt reports two 2026 supply-chain incidents. Hosted OpenRouter routes across hundreds of models, while llm-d places requests among self-hosted vLLM or SGLang replicas based on cache hits and load; it does not select providers. The excerpt reports an OpenRouter test where price, waiting-time and retry routing reduced median time to first token by 16.9% and 429 events by 67%, at about 6% higher cost per request. It recommends evaluating cost per completed task and accounting for cache locality, budgets, key custody and traceability.

### Source excerpt

TL;DR Routing teams are learning that token price is only part of what an answer costs. OpenRouter put numbers on it in October: in a...

## Google: LLMjacking attacks rack up $1 million in cloud AI charges

DevFeed: [Google: LLMjacking attacks rack up $1 million in cloud AI charges](<https://devfeed.tech/articles/google-llmjacking-attacks-rack-up-1-million-in-cloud-ai-charges-115538.md>)

Original publisher: [Read original article](<https://www.bleepingcomputer.com/news/security/google-llmjacking-attacks-rack-up-1-million-in-cloud-ai-charges/>)

Author: Bill Toulas

Published: 2026-11-03T18:51:04Z

Content type: news

Language: en

Sources: [BleepingComputer](<https://devfeed.tech/sources/bleepingcomputer.md>)

Topics: [Cybersecurity](<https://devfeed.tech/topics/cybersecurity.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [MFA](<https://devfeed.tech/topics/mfa.md>), [Cloud Logging](<https://devfeed.tech/topics/cloud-logging.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [business-email-compromise](<https://devfeed.tech/tags/business-email-compromise.md>), [charges](<https://devfeed.tech/tags/charges.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-ai](<https://devfeed.tech/tags/cloud-ai.md>), [cloud-infrastructure](<https://devfeed.tech/tags/cloud-infrastructure.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [compromise](<https://devfeed.tech/tags/compromise.md>), [computer-help](<https://devfeed.tech/tags/computer-help.md>), [computer-security](<https://devfeed.tech/tags/computer-security.md>), [computers](<https://devfeed.tech/tags/computers.md>), [costs](<https://devfeed.tech/tags/costs.md>), [credentials](<https://devfeed.tech/tags/credentials.md>), [cve](<https://devfeed.tech/tags/cve.md>), [cve-2026-40217](<https://devfeed.tech/tags/cve-2026-40217.md>), [cybercrime](<https://devfeed.tech/tags/cybercrime.md>), [databases](<https://devfeed.tech/tags/databases.md>), [google](<https://devfeed.tech/tags/google.md>), [infosec](<https://devfeed.tech/tags/infosec.md>), [infosec-computer-security](<https://devfeed.tech/tags/infosec-computer-security.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [linux](<https://devfeed.tech/tags/linux.md>), [llmjacking](<https://devfeed.tech/tags/llmjacking.md>), [mac](<https://devfeed.tech/tags/mac.md>), [malware](<https://devfeed.tech/tags/malware.md>), [malware-removal](<https://devfeed.tech/tags/malware-removal.md>), [security](<https://devfeed.tech/tags/security.md>), [spyware](<https://devfeed.tech/tags/spyware.md>), [support](<https://devfeed.tech/tags/support.md>), [tech-support](<https://devfeed.tech/tags/tech-support.md>), [technical-support](<https://devfeed.tech/tags/technical-support.md>), [virus](<https://devfeed.tech/tags/virus.md>), [virus-removal](<https://devfeed.tech/tags/virus-removal.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

Google's GTIG and Mandiant report that LLMjacking can generate more than $100,000 in fraudulent cloud charges in a day, with cumulative costs reaching $1 million in observed cases. Attackers find exposed LLM endpoints or steal credentials and API keys, then resell access through LLM mixers; request spikes can exceed 1,000 per second while providers bill the victim organization. The excerpt also describes credential theft, persistence attempts, exploitation of a LiteLLM vulnerability, and unauthorized proxy or agent deployments. Google recommends centralized secrets management and key rotation, least-privilege IAM, phishing-resistant MFA, visibility into unauthorized AI services, API usage and spending limits, and monitored cloud audit and inference logs.

### Source excerpt

Large language model hijacking attacks, also known as LLMjacking, are generating large fraudulent cloud charges, with some organizations losing more than $100,000 per day. [...]

## Google Warns LLMjacking Attacks Are Costing Some Enterprises Up to $1 Million in Cloud Charges

DevFeed: [Google Warns LLMjacking Attacks Are Costing Some Enterprises Up to $1 Million in Cloud Charges](<https://devfeed.tech/articles/google-warns-llmjacking-attacks-are-costing-some-enterprises-up-to-1-million-in-cloud-charges-114925.md>)

Original publisher: [Read original article](<https://thehackernews.com/2026/11/google-warns-llmjacking-attacks-are.html>)

Author: The Hacker News

Published: 2026-11-03T15:41:53Z

Content type: news

Language: en

Sources: [Cyber Newsroom](<https://devfeed.tech/sources/cyber-newsroom.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [cloud-infrastructure](<https://devfeed.tech/topics/cloud-infrastructure.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [Google](<https://devfeed.tech/topics/google.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-services](<https://devfeed.tech/tags/ai-services.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [charges](<https://devfeed.tech/tags/charges.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-security](<https://devfeed.tech/tags/cloud-security.md>), [computing](<https://devfeed.tech/tags/computing.md>), [credential-theft](<https://devfeed.tech/tags/credential-theft.md>), [cve](<https://devfeed.tech/tags/cve.md>), [cve-2026-40217](<https://devfeed.tech/tags/cve-2026-40217.md>), [cybercrime](<https://devfeed.tech/tags/cybercrime.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [google](<https://devfeed.tech/tags/google.md>), [hijacking](<https://devfeed.tech/tags/hijacking.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llmjacking](<https://devfeed.tech/tags/llmjacking.md>), [up](<https://devfeed.tech/tags/up.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

LLMjacking lets attackers use enterprise LLM accounts and leave the organization with the bill. Google reports access obtained through exposed or stolen API keys and cloud identity credentials, including Service Account keys, IAM credentials, GitHub tokens, and session tokens. Attackers may then search repositories and cloud environments for more credentials, escalate privileges, and create persistent access. Google observed individual incidents exceeding $100,000 in charges in a day and cumulative bills reaching $1 million before access was revoked. Monetization includes proxying model access and reselling it through illicit routing services; some attackers rotate short-lived service accounts to evade rate limits. Separately, Symantec and Carbon Black described a grey-market service that installs genuine OpenAI Codex and Anthropic Claude Code CLIs, then reconfigures them to route requests through its servers. The setup disables Codex's execution sandbox, replaces users' credentials with relay tokens, and disables telemetry and error reporting. The report says there is no evidence this service harvested prompts or altered responses.

### Source excerpt

Google Warns LLMjacking Attacks Are Costing Some Enterprises Up to $1 Million in Cloud Charges Google on Monday warned that threat actors are hijacking enterprise access to Large Language Models (LLMs) to fraudulently leverage expensive computing resources and AI services. The tactic is called ... Read more Published Date: Nov 03, 2026 (2 hours, 54 minutes ago) Vulnerabilities has been mentioned in this article. CVE-2026-40217

## Stolen AI access leaves companies with cloud bills of up to $1 million

DevFeed: [Stolen AI access leaves companies with cloud bills of up to $1 million](<https://devfeed.tech/articles/stolen-ai-access-leaves-companies-with-cloud-bills-of-up-to-1-million-113606.md>)

Original publisher: [Read original article](<https://www.helpnetsecurity.com/2026/11/03/llmjacking-stolen-ai-access-cloud-bills/>)

Author: Sinisa Markovic

Published: 2026-11-03T09:39:05Z

Content type: news

Language: en

Sources: [Help Net Security](<https://devfeed.tech/sources/help-net-security.md>)

Topics: [transfer stations](<https://devfeed.tech/topics/transfer-stations.md>), [Credential theft](<https://devfeed.tech/topics/credential-theft.md>), [credential-based attacks](<https://devfeed.tech/topics/credential-based-attacks.md>), [litellm](<https://devfeed.tech/topics/litellm.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [actor](<https://devfeed.tech/tags/actor.md>), [added](<https://devfeed.tech/tags/added.md>), [administrator](<https://devfeed.tech/tags/administrator.md>), [advice](<https://devfeed.tech/tags/advice.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [api](<https://devfeed.tech/tags/api.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cybercrime](<https://devfeed.tech/tags/cybercrime.md>), [financial-services](<https://devfeed.tech/tags/financial-services.md>), [google](<https://devfeed.tech/tags/google.md>), [google-threat-intelligence-group](<https://devfeed.tech/tags/google-threat-intelligence-group.md>), [gtig](<https://devfeed.tech/tags/gtig.md>), [llmjacking](<https://devfeed.tech/tags/llmjacking.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mandiant](<https://devfeed.tech/tags/mandiant.md>), [news](<https://devfeed.tech/tags/news.md>), [retail](<https://devfeed.tech/tags/retail.md>), [security](<https://devfeed.tech/tags/security.md>), [technology](<https://devfeed.tech/tags/technology.md>), [threat-intelligence](<https://devfeed.tech/tags/threat-intelligence.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Attackers use stolen cloud credentials or exposed AI API keys to activate or access companies' model services, then resell that capacity through underground "LLM Mixer" services. In reported cases, one operation made more than 500,000 model calls in under eight hours and cost the victim nearly $1 million; other bills exceeded $100,000 in a day. Unlike the slower resource drain described for cryptojacking, LLMJacking can create large charges within hours. Access methods included misconfigured service account keys, tokens harvested from repositories or pasted into a fake portal, and a LiteLLM gateway vulnerability. Mandiant advises defenders to monitor changes to AI accounts and services, since attackers in several incidents turned on services that had been disabled. The text describes investigated incidents and does not provide a broader prevention checklist.

### Source excerpt

Criminals who break into corporate cloud accounts are using the stolen access to run AI models at the victim's expense, in some cases racking up more than $100,000 in charges in a single day, Google Threat Intelligence Group (GTIG) and Mandiant researchers have found. Usage chart showing LLMJacking generating a spike in API requests (Source: Google) In 2026, Mandiant responded to several of these LLMJacking incidents, recorded in attacks on the financial services, retail, technology ... More -> The post Stolen AI access leaves companies with cloud bills of up to $1 million appeared first on Help Net Security.

[Next page](<https://devfeed.tech/topics/litellm.md?cursor=WyIyMDI2LTExLTAzVDA5OjM5OjA1KzAwOjAwIiwgIjE2ODU3N2FmLTk3YzEtNGJhMi1hMjE1LTU3ODBiN2E2MDQxNiJd>)