# Model Routing

Technique for selecting a large language model from a candidate pool for each request based on its requirements.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## ADAPT: Adaptive Decision and Abstention with Pool-Size-Aware Thresholds for Service Routing

DevFeed: [ADAPT: Adaptive Decision and Abstention with Pool-Size-Aware Thresholds for Service Routing](<https://devfeed.tech/articles/adapt-adaptive-decision-and-abstention-with-pool-size-aware-thresholds-for-service-routing-145583.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25754>)

Author: Manile Srun, Muntasir Adnan, Arslan Shaukat, Carlos C. N. Kuhn

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [added](<https://devfeed.tech/tags/added.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [classifier](<https://devfeed.tech/tags/classifier.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decision](<https://devfeed.tech/tags/decision.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [generative](<https://devfeed.tech/tags/generative.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [pool](<https://devfeed.tech/tags/pool.md>), [routing](<https://devfeed.tech/tags/routing.md>), [service](<https://devfeed.tech/tags/service.md>), [size](<https://devfeed.tech/tags/size.md>)

### AI overview

ADAPT adds an explicit fallback to retrieval-based service routing, allowing a query to abstain when no specialist appears suitable. It measures confidence using the gap between the top relevance score and the candidate score distribution. The authors report that this confidence shifts as the candidate pool grows, so a fixed abstention threshold becomes miscalibrated; ADAPT adjusts the threshold for pool size K without retraining. Across two specialist datasets, the abstract reports an effective accuracy-cost trade-off compared with generative LLM routing. These findings are limited to the abstract provided.

### Source excerpt

arXiv:2611.25754v1 Announce Type: new Abstract: Retrieval-based classifiers efficiently route queries to specialised services, but they typically select a candidate even when no suitable service exists. We introduce Adaptive Decision and Abstention with Pool-Size-Aware Thresholds for Service Routing (ADAPT), a lightweight routing architecture with an explicit fallback option that enables queries to abstain from specialist selection. ADAPT uses the separation between the highest relevance score and the candidate score distribution as routing confidence. Experiments across retrieval models and increasing numbers of services reveal a consistent empirical behaviour: the confidence distribution changes systematically with candidate-pool size K, making fixed abstention thresholds increasingly miscalibrated. ADAPT addresses this effect with a K-aware threshold that adapts as services are added without classifier retraining. Evaluation across two specialist datasets shows that this simple mechanism provides an effective accuracy-cost trade-off compared with generative LLM routing, supporting lightweight and dynamically extensible service-routing architectures.

## Perplexity Launches Orchestrator API for Failover and Model Routing

DevFeed: [Perplexity Launches Orchestrator API for Failover and Model Routing](<https://devfeed.tech/articles/perplexity-launches-orchestrator-api-for-failover-and-model-routing-144513.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/11/19/perplexity-orchestrator-api-failover-model-routing/>)

Author: Michal Sutter

Published: 2026-11-19T17:47:00Z

Content type: news

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [API](<https://devfeed.tech/topics/api.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [Chat completions endpoints](<https://devfeed.tech/topics/chat-completions-endpoints.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [billing](<https://devfeed.tech/tags/billing.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [failover](<https://devfeed.tech/tags/failover.md>), [for-devs](<https://devfeed.tech/tags/for-devs.md>), [google](<https://devfeed.tech/tags/google.md>), [launches](<https://devfeed.tech/tags/launches.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [models](<https://devfeed.tech/tags/models.md>), [new-releases](<https://devfeed.tech/tags/new-releases.md>), [open](<https://devfeed.tech/tags/open.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [openai](<https://devfeed.tech/tags/openai.md>), [orchestrator](<https://devfeed.tech/tags/orchestrator.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [preview](<https://devfeed.tech/tags/preview.md>), [route](<https://devfeed.tech/tags/route.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [staff](<https://devfeed.tech/tags/staff.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>), [xai](<https://devfeed.tech/tags/xai.md>)

### AI overview

Perplexity Orchestrator provides one endpoint for OpenAI, Anthropic, Google, xAI, and open-weight models, accepting OpenAI Chat Completions and Responses plus Anthropic Messages requests. It selects a serving provider by availability, capacity, and latency; failover keeps the requested model fixed, retries provider errors, rate limits, and timeouts, and does not bill attempts that fail before producing output. Invalid or over-context requests are not retried, and users cannot select a deployment. The preview `perplexity/auto` router chooses among described routes, falling back to a default model when no route matches or routing fails. Perplexity reports a 63-task test where refusal routing raised completion from 38 to 41 tasks, with higher estimated cost; results may vary by workload. Requests use the answering model's published token rate, with no additional platform fee, while model access depends on cumulative credit purchases. The excerpt reports reliability comparisons without success-rate percentages and describes provider speed differences.

### Source excerpt

Perplexity's Orchestrator API serves OpenAI, Anthropic, Google, xAI and open-weight models through 1 endpoint, with same-model provider failover, unified billing and a preview perplexity/auto router that sends each request to a described route. The post Perplexity Launches Orchestrator API for Failover and Model Routing appeared first on MarkTechPost.

## A Differential Theory of Quality-Resource Trade-offs for AI Model Selection: The Multiresource Frontier and Its Minimax Price

DevFeed: [A Differential Theory of Quality-Resource Trade-offs for AI Model Selection: The Multiresource Frontier and Its Minimax Price](<https://devfeed.tech/articles/a-differential-theory-of-quality-resource-trade-offs-for-ai-model-selection-the-multiresource-frontier-and-its-minimax-price-143377.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23324>)

Author: Dmitry Ignatov

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [quality](<https://devfeed.tech/tags/quality.md>), [resource](<https://devfeed.tech/tags/resource.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theory](<https://devfeed.tech/tags/theory.md>), [trade](<https://devfeed.tech/tags/trade.md>)

### AI overview

The paper proposes a differential framework for comparing AI model quality-resource frontiers. It represents each frontier through the relation between effective active resource channels (squared slope over curvature) and remaining log-error depth, separating intrinsic shape from the resource axis's scale and origin. Complete monotonicity and divisibility yield inequalities and frontier bounds. For N resources with one active channel each, it derives an operating rule of N_eff = 1 in delivered units and minimax prices λ*_i = α_i/(N+1), with worst-case regret ln(N+1) nats. The authors say this enables resource-cost comparisons without hand-chosen exchange rates. They report evaluating the theory on 117 published frontiers and implementing nn-rank to derive prices from measured quality and cost, rank candidates, and report supported choices and minimax regret.

### Source excerpt

arXiv:2611.23324v1 Announce Type: cross Abstract: Comparing AI models across resource budgets yields quality-resource frontiers relating error to latency, parameter count, training compute, and energy. Offset power laws summarize such frontiers, but their fitted parameters reveal neither the price-independent geometry nor the manner in which trade-offs extend across several resource channels. This paper addresses both gaps by developing a differential theory in which this structure is made explicit. Each frontier admits a price-independent plane-curve representation relating the effective number of active resource channels, $N_{\mathrm{eff}} = s^2/\kappa$ (squared slope over curvature), to the remaining log-error depth, $\Delta S = \ln(\varepsilon/\varepsilon_{\infty})$. This representation separates intrinsic shape from the scale and origin of the resource axis. Complete monotonicity and divisibility yield differential inequalities and frontier bounds. When each of $N$ resources contributes one active channel, the theory gives the parameter-free operating rule $N_{\mathrm{eff}} = 1$ in delivered units and derives minimax resource prices $\lambda^{\star}_i = \alpha_i/(N+1)$ with an exact worst-case regret of $\ln(N+1)$ nats. The results make it possible to interpret measured frontiers and compare resource costs without hand-chosen exchange rates. The theory is evaluated on 117 published frontiers and implemented in the open-source package nn-rank, which derives prices from measured quality and cost, ranks candidates, and reports supported choices and minimax regret: https://github.com/ABrain-One/nn-rank

## What Does an Agent Need? Bundle-Aware Agent Resource Composition

DevFeed: [What Does an Agent Need? Bundle-Aware Agent Resource Composition](<https://devfeed.tech/articles/what-does-an-agent-need-bundle-aware-agent-resource-composition-139949.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22030>)

Author: Yunxiao Shi (Australian National University), Tingwei Chen (University of Technology Sydney), Dannong Xu (Australian National University), Ahad N. Zehmakan (Australian National University), Jialie She

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [beam](<https://devfeed.tech/tags/beam.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [complete](<https://devfeed.tech/tags/complete.md>), [composition](<https://devfeed.tech/tags/composition.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [execution](<https://devfeed.tech/tags/execution.md>), [experience](<https://devfeed.tech/tags/experience.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fine](<https://devfeed.tech/tags/fine.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generator](<https://devfeed.tech/tags/generator.md>), [generators](<https://devfeed.tech/tags/generators.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [resource](<https://devfeed.tech/tags/resource.md>)

### AI overview

AgentComposition jointly selects a backbone model and tool-skill bundle for a task, addressing the limits of generators tied to a training catalog and methods that score resources independently. Its retrieval-grounded generator combines candidate resources and past configurations through constrained beam search, while a bundle-aware critic ranks complete configurations using an LLM teacher and successful historical configurations as execution proxies; it can also adapt to execution feedback. On AgentSelect queries, the method recovered the full reference configuration within ten candidates 67.73% of the time, nearly twice the strongest baseline. The critic raised top-agent execution scores from 7.58 to 8.37 out of 10, and feedback raised them to 8.78. Four target-domain queries adapted the generator to SkillsBench, where the method led baselines. Online recommendations also outperformed the strongest baselines under an LLM judge and native verifiers.

### Source excerpt

arXiv:2611.22030v1 Announce Type: new Abstract: An agent resource's value depends on its backbone and companion resources and is known only after execution. Token-based generators remain tied to their training catalog, while independent resource scoring misses complementarities and reused bundles target other tasks. We formulate Task-Conditioned Agent Resource Composition (AgentComposition): jointly selecting a backbone model and a tool-skill bundle to maximize end-to-end execution utility. Our Retrieval-Grounded Bundle-Aware Agent Generation retrieves candidate resources and past configurations of similar tasks. A fine-tuned generator reads resource descriptions and recombines candidates through constrained beam search, allowing new resources without retraining. A bundle-aware critic ranks complete configurations using a large language model (LLM) teacher and historically successful configurations as execution proxies, and adapts to execution feedback when available. We evaluate reference recovery, offline execution with judged and native rewards, and online recommendation with real execution, building AgentSelect-Exec with 5,060 executed configurations. Our method recovers the complete reference configuration within ten candidates for 67.73% of AgentSelect queries, nearly twice the strongest baseline, and leads all baselines on SkillsBench. The proxy-trained critic raises the top-ranked agent's execution score from 7.58 to 8.37 out of 10; execution feedback increases it to 8.78, above the strongest baselines. Four target-domain queries suffice to adapt the generator to SkillsBench. Agents recommended online outperform the strongest baselines under both an LLM judge and native verifiers. These results support learning and transferring joint resource composition from task experience.

## LatentRouter: Can We Choose the Right Multimodal Large Language Model Before Seeing Its Answer?

DevFeed: [LatentRouter: Can We Choose the Right Multimodal Large Language Model Before Seeing Its Answer?](<https://devfeed.tech/articles/latentrouter-can-we-choose-the-right-multimodal-large-language-model-before-seeing-its-answer-137254.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.11301>)

Author: Xueqi Cheng, Yushun Dong

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [bench](<https://devfeed.tech/tags/bench.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [latency](<https://devfeed.tech/tags/latency.md>), [layout](<https://devfeed.tech/tags/layout.md>), [mmr](<https://devfeed.tech/tags/mmr.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [token](<https://devfeed.tech/tags/token.md>), [utility](<https://devfeed.tech/tags/utility.md>)

### AI overview

LatentRouter selects among multimodal large language models by predicting each candidate's counterfactual utility for an image-question query, matching learned query capsules to model capability tokens through latent communication. A distributional outcome head predicts model-specific quality, and a bounded capsule correction refines close choices. Its utility policy supports performance-focused or performance-cost routing and changing candidate pools via shared per-model scoring and availability masking. On MMR-Bench and VL-RouterBench, it reportedly outperforms fixed-model, feature-level, and learned-router baselines, with strongest gains on visually, layout, or reasoning-dependent tasks; analyses identify latent communication as the main contributor.

### Source excerpt

arXiv:2605.11301v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.

## Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints

DevFeed: [Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints](<https://devfeed.tech/articles/beyond-leaderboard-ranks-selecting-software-engineering-agents-under-budget-constraints-138019.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21515>)

Author: Srinath Srinivasan, Tim Menzies

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>), [harvestbench](<https://devfeed.tech/topics/harvestbench.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [agents](<https://devfeed.tech/tags/agents.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [budget](<https://devfeed.tech/tags/budget.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [execution](<https://devfeed.tech/tags/execution.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>)

### AI overview

Leaderboard rank alone may not identify the best-value software engineering agent: accuracy gaps are often statistically unresolved, and reported costs can diverge from standardized token-based estimates by up to 11.7x. In 58,197 HAL run-task records across seven benchmarks, 27 of 35 adjacent steps on accuracy-cost Pareto frontiers were statistically unresolved. On SWE-bench, successive halving met the study's uncertainty-aware selection target using 6.6% of exhaustive evaluation spend when task costs were known in advance; it had the lowest cost on six benchmarks and tied on one. Its advantage narrowed when costs were unknown before execution or exact-best selection was required. The authors also find that selective task evaluation can bias full-benchmark cost estimates, motivating evaluation that accounts for uncertainty and validates costs.

### Source excerpt

arXiv:2611.21515v1 Announce Type: new Abstract: Teams that adopt LLM agents for issue resolution must decide which agent to pay for, and how much to spend finding out. Leaderboard ranks give weak evidence for this decision. Accuracy differences may be statistically unresolved, reported costs may be inconsistent, and evaluating candidate agents can itself be expensive. We investigate budget-constrained agent selection using 58,197 run--task records extracted from the Holistic Agent Leaderboard (HAL), focusing on SWE-bench and six additional benchmarks. Across seven benchmarks, 27 of 35 steps along the accuracy--cost Pareto frontiers are statistically unresolved. On SWE-bench, a run costing $34.16 trails the $170.78 leader by 14.6 percentage points, but the available tasks cannot statistically distinguish their accuracy. Reported costs also differ from standardized token-based estimates by up to 11.7x, while selective task evaluation can bias estimates of full-benchmark execution cost. We therefore compare evaluation strategies under explicit evaluation budgets and deployment-cost caps. Under advance knowledge of task costs, successive halving reaches our uncertainty-aware selection target on SWE-bench using 6.6% of exhaustive evaluation spend, achieving the lowest cost on six of seven benchmarks and tying on the seventh. Its advantage narrows when task costs are unknown before execution or exact-best selection is required. Our findings motivate decision-oriented agent evaluation that accounts for accuracy uncertainty, validates execution costs, and compares complex selection methods against simple adaptive allocation.

## Decision-1.1 now available on AI Gateway

DevFeed: [Decision-1.1 now available on AI Gateway](<https://devfeed.tech/articles/decision-1-1-now-available-on-ai-gateway-138549.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/decision-1-1-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-17T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>)

Tags: [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-responses](<https://devfeed.tech/tags/ai-responses.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [choice](<https://devfeed.tech/tags/choice.md>), [cli](<https://devfeed.tech/tags/cli.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decisions-api](<https://devfeed.tech/tags/decisions-api.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [playground](<https://devfeed.tech/tags/playground.md>), [python-sdk](<https://devfeed.tech/tags/python-sdk.md>), [routing](<https://devfeed.tech/tags/routing.md>), [score](<https://devfeed.tech/tags/score.md>), [structured](<https://devfeed.tech/tags/structured.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

Microsoft AI's Decision-1.1 is available on AI Gateway as an updated decision model, described as more accurate, better calibrated, and faster than Decision-1. It returns structured Boolean probabilities, category choices with probabilities, or scores on a defined scale, and can answer multiple questions from one input. Its calibrated probabilities can support thresholds for acting, reviewing, or escalating cases. It accepts the same request format as Decision-1; switching requires changing the model ID to `microsoft/decision-1.1`. It can be called through the AI SDK, OpenAI-compatible Decisions API, or AI CLI. The excerpt's examples require `ai` 7.0.128+ for TypeScript or OpenAI Python SDK 3.26.0+ for Python. Requests appear in logs, count toward budgets, and support decision fallbacks.

### Source excerpt

Decision-1.1 from Microsoft AI is now available on AI Gateway. It is more accurate, better calibrated, and faster than Decision-1. The decision model returns structured answers instead of free-form text for tasks such as routing support requests, choosing which model should handle a request, selecting the next step in a workflow, or grading AI responses. Boolean: the probability that a condition is true Choice: a selected category and probabilities for each option Score: a numerical rating on a scale you define Multiple questions can share the same input in one request. Better calibration means the model's probabilities track how often it is right, so your application can set thresholds for when to act, flag a case for review, or escalate. Decision-1.1 accepts the same request format as Decision-1. To switch, change the model ID to microsoft/decision-1.1. Call it through the AI SDK, the OpenAI-compatible Decisions API, or the AI CLI. The examples below require ai 7.0.128 or later for TypeScript, or the OpenAI Python SDK 3.26.0 or later for Python. Decision requests appear in request logs, count toward budgets, and support decision fallbacks. Try Decision-1.1 in the model playground, or see the decision models documentation for request formats and question types. Read more

## AI Gateway now supports Private Inference

DevFeed: [AI Gateway now supports Private Inference](<https://devfeed.tech/articles/ai-gateway-now-supports-private-inference-136322.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/ai-gateway-now-supports-private-inference>)

Author: Jerilyn Zheng

Published: 2026-11-16T07:47:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [inference-providers](<https://devfeed.tech/topics/inference-providers.md>), [Chat completions endpoints](<https://devfeed.tech/topics/chat-completions-endpoints.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Self Hosted AI Tools](<https://devfeed.tech/topics/self-hosted-ai-tools.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [apis](<https://devfeed.tech/tags/apis.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [chat](<https://devfeed.tech/tags/chat.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [inference](<https://devfeed.tech/tags/inference.md>), [logs](<https://devfeed.tech/tags/logs.md>), [messages](<https://devfeed.tech/tags/messages.md>), [model](<https://devfeed.tech/tags/model.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [observability](<https://devfeed.tech/tags/observability.md>), [private](<https://devfeed.tech/tags/private.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [responses](<https://devfeed.tech/tags/responses.md>), [sdks](<https://devfeed.tech/tags/sdks.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

Vercel AI Gateway's Private Inference lets teams route fine-tuned, self-hosted, or prerelease models through the same authentication, SDKs, APIs, and observability used for catalog models. Add an endpoint as a private provider with its base URL, API format, and credentials, then register its models; setup is also available through Vercel CLI 65.16.0 or later. Requests use a `private/<slug>` model ID through Chat Completions, Responses, or Messages, with AI Gateway translating requests to the endpoint's format. Private models are limited to the Vercel team and excluded from the public catalog; credentials are encrypted, and Zero Data Retention can be enabled per provider. Logs and reporting include private model traffic, with charts for requests, tokens, and time to first token. A catalog model can be configured as a fallback if a private endpoint fails. The excerpt points to documentation for supported model types, requirements, and billing details.

### Source excerpt

Private Inference connects your own model endpoints to AI Gateway. Enterprises running fine-tuned, self-hosted, or prerelease models used to need separate keys, client code, and logs for them. Those models can now run through the same authentication, SDKs and APIs, and observability as catalog models, so every model your team uses is managed in one place. Use Private Inference to: Switch between private and catalog models: Call private models through Chat Completions, Responses, or Messages. AI Gateway translates each request to your endpoint's format, so switching only changes the model ID. Keep models scoped to your team: Only your Vercel team can view or call private models, and they aren't listed in the public catalog. Credentials are stored encrypted, and you can enable Zero Data Retention per provider. Connect coding agents: Use private models in Claude Code, Codex, Cursor, and other coding agents connected to AI Gateway. Monitor traffic together: View private model requests alongside catalog traffic in logs, trace drains, and custom reporting. Observability charts break down requests, tokens, and time to first token by private model. Configure fallbacks: Optionally fall back to a catalog model if your private endpoint fails. To get started, open the Private Inference page and add your endpoint as a private provider. Enter its base URL, select its API format, and add its credentials. Then add the models it serves. You can also register the provider and model with the Vercel CLI, version 65.16.0 or later: Call the model using its private/<slug> ID: Private model requests appear in AI Gateway logs alongside requests to catalog models. type: embedded-entry-inline id: 1GzjEiRrj3NaTriLhtBYIX Read the Private Inference documentation for supported model types, requirements, and billing details. Read more

## How Do Companies Build Their Own AI? A Look Inside Enterprise AI Architecture

DevFeed: [How Do Companies Build Their Own AI? A Look Inside Enterprise AI Architecture](<https://devfeed.tech/articles/how-do-companies-build-their-own-ai-a-look-inside-enterprise-ai-architecture-132381.md>)

Original publisher: [Read original article](<https://medium.com/google-developer-experts/how-do-companies-build-their-own-ai-a-look-inside-enterprise-ai-architecture-df815f44f896?source=rss----a67bd6fa7d58---4>)

Author: Geeta Kakrani

Published: 2026-11-13T14:23:08Z

Content type: article

Language: en

Sources: [Google Developer Experts](<https://devfeed.tech/sources/google-developer-experts-medium.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agent-guardrails](<https://devfeed.tech/tags/agent-guardrails.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-architecture](<https://devfeed.tech/tags/ai-architecture.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [enterprise-ai-architecture](<https://devfeed.tech/tags/enterprise-ai-architecture.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [rags](<https://devfeed.tech/tags/rags.md>)

### AI overview

Production enterprise AI requires multiple engineering layers beyond an LLM and chat interface. The article explains using RAG to ground answers in company documents, and tool calling through MCP to let agents interact with HR, Jira, Slack and other systems. Authentication and RBAC must enforce each user's permissions at the backend; guardrails such as injection detection and PII filtering add protections but do not replace authorization. Semantic caching and model routing can reduce cost and latency, with care for permissions, freshness and quality. Observability tracks retrieval, model calls, tools and execution traces; evaluations test accuracy, faithfulness, retrieval, tool use, security, cost and regressions after changes. Together these layers support safer, more reliable production systems.

### Source excerpt

Building an AI agent is easy. Making it reliable, secure, cost-efficient, and ready for thousands of employees is the real engineering challenge.Imagine your company has deployed an AI assistant for all its employees. An employee opens the chat interface and asks: "What is our company's leave policy?" The AI responds confidently: "You get 30 paid leaves and unlimited sick leave." The answer sounds professional. The employee trusts it and plans their vacation accordingly. But there is a problem. Your company never defined this policy. The AI simply invented the answer. This is called hallucination, and it is one of the biggest challenges when deploying Large Language Models (LLMs) in real business environments. Now imagine another situation. Your company has connected this AI assistant to its HR system, Jira, Slack, and internal databases. Employees can ask questions, apply for leave, raise IT tickets, and access company information. It looks impressive. But what happens when an intern asks for the CTO's or Senior Manager's salary? What happens when someone tries to manipulate the agent into revealing confidential information? What happens when 10,000 employees start using it every day and your API bill increases dramatically? These are not theoretical problems. They are real engineering challenges that appear when we move from AI prototypes to production systems. Building an enterprise AI agent is not just about selecting a powerful model and connecting it to a chatbot interface. It requires multiple layers of engineering. Let's understand these layers through a practical example: building an AI assistant for a company. 1. RAG: Stop Your AI from Inventing Company Policies The first problem is knowledge. An LLM has been trained on large amounts of publicly available information, but it does not automatically know your company's internal documents. It does not know: Your employee leave policy. Your work-from-home guidelines. Your reimbursement process. Your internal I

## Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery

DevFeed: [Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery](<https://devfeed.tech/articles/ballad-bandit-based-llm-routing-for-automated-heuristic-discovery-132325.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17241>)

Author: Samidha Verma, Ankit Anand, Sayan Ranu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [api](<https://devfeed.tech/tags/api.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [combinatorial-optimization](<https://devfeed.tech/tags/combinatorial-optimization.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dag](<https://devfeed.tech/tags/dag.md>), [decision](<https://devfeed.tech/tags/decision.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [genealogy](<https://devfeed.tech/tags/genealogy.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [np-hard](<https://devfeed.tech/tags/np-hard.md>), [online](<https://devfeed.tech/tags/online.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [policy](<https://devfeed.tech/tags/policy.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

Ballad adaptively routes calls among four LLMs during automated heuristic discovery, addressing the fact that a model's usefulness can shift as outputs seed later search steps and rewards become entangled across the heuristic population. Its policy favors rare, high-scoring heuristics, propagates credit through the population's genealogy as a DAG, and discounts stale observations. Across five NP-hard optimization problems, the authors report that Ballad matched the best individual model in hindsight and sometimes surpassed every individual model, while cutting API costs by up to 65.6% compared with always using the strongest model. The abstract reports no consistent heuristic-quality advantage for larger, costlier models.

### Source excerpt

arXiv:2611.17241v1 Announce Type: new Abstract: Modern automated heuristic discovery (AHD) systems increasingly rival hand-crafted heuristics by leveraging Large Language Models (LLMs) to iteratively mutate and refine solutions. A natural assumption is that more powerful and more expensive models consistently yield better heuristics. We challenge this empirically: across multiple NP-hard combinatorial optimization problems, we find no consistent advantage for larger, costlier models. This raises a deeper question: given a pool of LLMs spanning frontier and freely hosted or open-weight models, how should one adaptively select which model to invoke at each step of the search? The challenge is non-trivial. The optimal LLM may shift over the course of a run, and because one LLM's output seeds the context for subsequent calls, each model's reward signal is non-stationary and entangled across the heuristic population, thus violating the independence assumptions of classical online decision-making settings. We address this through Ballad (Bandit-based LLM Routing for Automated Heuristic Discovery), an online bandit algorithm that learns to orchestrate a pool of LLMs into an ensemble that matches or exceeds what any soloist LLM achieves alone. This orchestration is powered by an LLM selection policy that favors rare, high-scoring heuristics over stable averages, a DAG-based credit propagation mechanism through the genealogy of the heuristic population, and discounting of stale observations to remain attuned to each model's current utility. Across five NP-hard combinatorial optimization problems and a pool of four models of varying capability and cost, Ballad not only matches the best individual LLM in hindsight but, in several cases, surpasses every individual model while reducing API cost by up to 65.6% relative to always using the strongest model.

## Smart Model Routing for Order-of-Magnitude LLM Cost Reduction

DevFeed: [Smart Model Routing for Order-of-Magnitude LLM Cost Reduction](<https://devfeed.tech/articles/smart-model-routing-for-order-of-magnitude-llm-cost-reduction-131537.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15911>)

Author: Hesham Elbakoury, Deep Shah, Ankur Sharma

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [cascade](<https://devfeed.tech/tags/cascade.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failures](<https://devfeed.tech/tags/failures.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-cost-reduction](<https://devfeed.tech/tags/llm-cost-reduction.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [production](<https://devfeed.tech/tags/production.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>)

### AI overview

Smart model routing can cut blended LLM inference costs by roughly an order of magnitude in the paper's numerical example, under its stated pricing and traffic assumptions; the net savings account for router overhead. The paper frames routing as sending each query to the cheapest model expected to answer adequately, and derives a cost-quality framework covering threshold policies, calibration versus discrimination, cascade break-even conditions, and sensitivity to price ratios and traffic skew. It compares the example with published systems reporting roughly 2x to frontier-level accuracy at one-fifth the cost. The analysis is mainly theoretical, with a GSM8K evaluation of the cascade framework; broader evaluation across tasks and model families is future work. Production savings may be undermined by routing errors, segment-level quality harms hidden by aggregate constraints, calibration drift, prompt injection into routing signals, tier outages, or multi-provider price variation.

### Source excerpt

arXiv:2611.15911v1 Announce Type: new Abstract: Frontier large language models (LLMs) deliver the strongest response quality but carry per-token prices often more than an order of magnitude above those of smaller, task-adequate models. Smart model routing addresses this mismatch by assigning each query to the cheapest model expected to answer it adequately, rather than sending every request to the most capable, most expensive endpoint. This paper develops a formal cost-quality optimization framework for routing -- including a derived optimal threshold-policy structure, a calibration-versus-discrimination distinction, a cascade breakeven condition, and a sensitivity analysis over price ratio and traffic skew -- situates that framework against the principal routing architectures and the signals used to estimate query difficulty, and works through a numerical example showing how a calibrated router can reduce blended inference cost by roughly an order of magnitude relative to an always-frontier deployment under specific, stated pricing and traffic assumptions, reporting both the gross reduction from routing policy alone and the net figure with the router's own overhead priced in. We situate this figure against the range reported by published routing systems (roughly 2x to frontier-level accuracy at one-fifth the cost), and close with the practical risks -- under- and over-routing, aggregate quality constraints masking segment-level harm, calibration without discrimination, calibration drift, prompt injection into routing signals, and operational failures such as tier outages and multi-provider price variation -- that determine whether the savings materialize in production. The analysis is primarily analytical, complemented by a GSM8K evaluation of the cascade framework (Section 5), validated (Section 5.2); extending the evaluation across tasks and model families is future work.

## What is Model Routing in AI?

DevFeed: [What is Model Routing in AI?](<https://devfeed.tech/articles/what-is-model-routing-in-ai-133713.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/what-is-model-routing-in-ai/>)

Author: Matthew Duffin

Published: 2026-11-13T00:41:35Z

Content type: tutorial

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-costs](<https://devfeed.tech/tags/ai-costs.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-model-routing](<https://devfeed.tech/tags/ai-model-routing.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

Model routing sends each request to a model suited to its difficulty, reducing cost while reserving stronger models for uncertain cases. The guide distinguishes hand-written rules, classifier-based intelligent routing, confidence-based cascades, and fallback routing for availability. Its JavaScript cascade sends high-confidence classifications to TypeSafe's Jev and escalates uncertain ones to Claude Opus 5.5, falling back to Jev if Claude fails. On 200 Banking77 messages, a 0.8 confidence threshold routed about one in ten to Opus and cut cost by 89%, with accuracy roughly 0.5-1.5 points below Opus depending on replay or live run; a 0.95 threshold matched Opus accuracy with 68% lower cost. Escalated requests took about 2.1 seconds versus 227 ms median for requests kept on Jev. The guide cautions that confident errors remain possible, escalation can also introduce errors, and thresholds should be evaluated on representative data and retested as models change.

### Source excerpt

null Continue reading What is Model Routing in AI? on SitePoint.

## Mubit + Neon AI Gateway: an agent that cuts its own frontier-model tokens by 73%

DevFeed: [Mubit + Neon AI Gateway: an agent that cuts its own frontier-model tokens by 73%](<https://devfeed.tech/articles/mubit-neon-ai-gateway-an-agent-that-cuts-its-own-frontier-model-tokens-by-73-132646.md>)

Original publisher: [Read original article](<https://neon.com/blog/mubit-neon-ai-gateway>)

Author: Carlota Soto

Published: 2026-11-12T09:00:00Z

Content type: article

Language: en

Sources: [Neon Docs](<https://devfeed.tech/sources/blog-neon-docs.md>)

Topics: [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Frontier Model](<https://devfeed.tech/topics/frontier-model.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [community](<https://devfeed.tech/tags/community.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [model](<https://devfeed.tech/tags/model.md>), [neon](<https://devfeed.tech/tags/neon.md>), [product](<https://devfeed.tech/tags/product.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

In the described data analyst agent benchmark, routing routine calls through Mubit cut gpt-5.5 token use by 73% and response time by 40% without losing accuracy, with a tuned model handling about two thirds of calls. Mubit groups similar calls, trains a small model after at least 25 good replies, and gates it through held-out exams and shadow, canary, and promotion stages; it checks quality after promotion and falls back to Neon AI Gateway if quality drops. The tuned model handles tasks such as listing tables, reading definitions, and formatting answers, while gpt-5.5 retains SQL generation: the small model matched its results on routine steps but achieved the same SQL result only 8-10% of the time. Verified-run lessons capture schema mappings, SQL, and pitfalls, and are tied to a schema version; on changed-schema benchmark runs, gpt-5.5 alone got 7 of 30 right, compared with 14 using lessons. These are results from the template's benchmark, not a general performance guarantee. The agent runs on Neon; Mubit trains and serves the tuned model on its infrastructure.

### Source excerpt

Mubit is middleware for AI agents. It sits between your agent and the model with a one-line endpoint change, learns from every call, and moves recurring steps to small models tuned on your own traffic, while the frontier model keeps the hard ones. There's nothing extra to host...

## Deletion Sensitivity Is a Property of the Model: A Three-Condition Measurement across Seven Open Reasoning Models

DevFeed: [Deletion Sensitivity Is a Property of the Model: A Three-Condition Measurement across Seven Open Reasoning Models](<https://devfeed.tech/articles/deletion-sensitivity-is-a-property-of-the-model-a-three-condition-measurement-across-seven-open-reasoning-models-128274.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13223>)

Author: Hanzhang Jia, Liheng Zeng, Zhiyu Yang, Yi Gao, Bo Ma

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [condition](<https://devfeed.tech/tags/condition.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [property](<https://devfeed.tech/tags/property.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-models](<https://devfeed.tech/tags/reasoning-models.md>)

### AI overview

The study reports that removing full reasoning history from multi-turn tool sessions can change task performance substantially, with effects ranging from −4.4 to +51.3 points across seven open-weight reasoning models, 794 tasks, and three seeds; both endpoints excluded zero. For the one model tested across deletion levels from 0% to 100%, the measured effect varied by at most 1.5 points. The authors argue sensitivity follows the model across deployments, while retrained or re-released models should be treated as new models. They propose a two-run screen--one with history retained and one with history deleted--and using the result as prior information in routing and aggregation. This overview is based on the supplied abstract and excerpt.

### Source excerpt

arXiv:2611.13223v1 Announce Type: cross Abstract: Reasoning models write a reasoning trace at every turn of a multi-turn tool session, and later turns read this history. Across 94k ShareGPT conversations, 73 percent of user interactions span multiple turns. Upstream providers bind each trace to its issuer, so a request forwarded across issuers arrives with its history deleted, and the task cost of this stripping remains unmeasured. To the best of our knowledge, this paper presents the first direct, blind, multi-seed measurement of full reasoning-history deletion in multi-turn tool calling, through three rendering conditions, seven open-weight reasoning models, three seeds, 794 tasks, and the official checker. The same deletion spans -4.4 to +51.3 points with both ends excluding zero, and for the one model re-dosed from 0 to 100 percent deletion, the reading varies by at most 1.5 points. Deletion sensitivity is a model-level property. The reading follows the model itself and stays constant across deployments, and a retrained or re-released model is a new model that a two-cell screen, one retention run and one deletion run, re-reads for the cost of two runs. Deletion sensitivity is thus a measurable property of the model, and the reading must enter routing and aggregation as prior information rather than persist as an invisible default cost.

## ARoute: Task-Aligned Batch and Streaming Model Routing

DevFeed: [ARoute: Task-Aligned Batch and Streaming Model Routing](<https://devfeed.tech/articles/aroute-task-aligned-batch-and-streaming-model-routing-128862.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13209>)

Author: Daofu Zhang, Qingcheng Zeng, Chunwei Liu, Fuheng Zhao

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [livecodebench](<https://devfeed.tech/tags/livecodebench.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [online](<https://devfeed.tech/tags/online.md>), [processes](<https://devfeed.tech/tags/processes.md>), [query](<https://devfeed.tech/tags/query.md>), [question](<https://devfeed.tech/tags/question.md>), [quota](<https://devfeed.tech/tags/quota.md>), [request](<https://devfeed.tech/tags/request.md>), [router](<https://devfeed.tech/tags/router.md>), [routers](<https://devfeed.tech/tags/routers.md>), [routing](<https://devfeed.tech/tags/routing.md>), [select](<https://devfeed.tech/tags/select.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

ARoute selects a model for each request before inference to maximize task-specific accuracy under a cost budget. Its batch variant routes requests with the greatest expected benefit to the stronger model when the batch is known in advance. Its streaming variant processes requests online, calibrating a threshold on a small validation set and adjusting it as spending accumulates; the authors report a finite-sample guarantee on expected normalized spending. Across MMLU-Pro, LiveCodeBench, SuperGPQA, and two SWAN-based value-filling tasks, ARoute-Batch had the highest mean accuracy in all 15 task-cost settings, beating the strongest baseline by 4.1 percentage points on average and up to 11.9 points. ARoute completed batches up to 2.8 times faster than cascades.

### Source excerpt

arXiv:2611.13209v1 Announce Type: new Abstract: Language models increasingly power data-intensive applications, from semantic data processing to question answering and agentic workflows. Using frontier models for every request is prohibitively expensive, while relying only on smaller models reduces accuracy. We study model routing under a user-specified cost budget, aiming to maximize accuracy under task-specific correctness criteria. Existing routers fall short in two ways: pretrained routers rely on generic preference signals that may not reflect a task's own correctness criteria or a new model pair, and model cascades run the weaker model first on every request, adding sequential latency. We present ARoute, a task-aligned router that learns cost and accuracy signals to select a model for each request before inference. ARoute has two variants: ARoute-Batch operates on a batch of requests known in advance, routing requests with the greatest expected benefit to the stronger model subject to the budget, whereas ARoute-Stream processes requests online, using a threshold calibrated on a small validation set to satisfy the cost quota and dynamically adjusting it as the budget is consumed. For ARoute-Stream, we provide a finite-sample guarantee on expected normalized spending. We evaluate ARoute on MMLU-Pro, LiveCodeBench, SuperGPQA, and two value-filling tasks built from SWAN semantic query benchmark against other state-of-the-art approaches. At matched cost, ARoute-Batch achieves the highest mean accuracy in all 15 task-cost settings, improving over the strongest baseline by 4.1 percentage points on average and by up to 11.9 points. ARoute also completes a batch up to 2.8x faster than cascades.

## SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving

DevFeed: [SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving](<https://devfeed.tech/articles/shadowprefill-keeping-your-next-model-warm-for-per-step-model-routing-in-agentic-llm-serving-127799.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13176>)

Author: Haoran Qiu, Rohan Mahapatra, Rodrigo Fonseca, Mark Russinovich

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-benchmark](<https://devfeed.tech/tags/agentic-benchmark.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [context](<https://devfeed.tech/tags/context.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [energy](<https://devfeed.tech/tags/energy.md>), [host-memory](<https://devfeed.tech/tags/host-memory.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [routing](<https://devfeed.tech/tags/routing.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

Model switches in per-step LLM routing lose prefix-cache reuse because KV caches are model-specific, lowering prefix hit rates from 80-89% on same-model calls to 44-65% after a switch and making switch calls 1.5-3.2x slower. SHADOWPREFILL uses otherwise idle capacity to prefill the session context on the unselected model, exploiting in-flight requests' SLO slack, spilling shadow KV to host memory, and reverting to a cold switch if the shadow is incomplete. In evaluations routing 24 model pairs across five agentic benchmark suites, it reduced switch-call latency by 8.5-21.4%, LLM-call latency by 3.7-17.7%, and routed token cost by 22.5-46.1%; normal traffic retained its SLO, while energy per request rose 0.4%.

### Source excerpt

arXiv:2611.13176v1 Announce Type: new Abstract: Per-step model routers send each LLM call of an agent to the cost-optimal model dynamically within each session. However, every model switch discards prefix-cache reuse: KV caches are per model, so the new model must prefill the relevant context before it emits a token. Across representative agentic workloads, including a production coding-agent trace with a mean context of 63K tokens, the prefix hit rate falls from 80-89% on same model calls to 44-65% after a switch, and a switch call takes 1.5-3.2x as long. We present SHADOWPREFILL, which prefills the session's context on the model the router has not chosen, in capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. SHADOWPREFILL harvests the per-iteration SLO slack of requests in flight, protects routed calls, spills shadow KV to host memory, and falls back to a cold switch whenever the shadow is incomplete. With LLMRouter routing 24 model pairs over five agentic benchmark suites, SHADOWPREFILL cuts switch-call latency by 8.5-21.4%, LLM-call latency by 3.7-17.7%, and routed token cost by 22.5-46.1%, while normal traffic keeps its SLO and energy per request rises by 0.4%.

## Self-Indexing Latent Memories for Access-Aware Continual Instruction Tuning of LLMs

DevFeed: [Self-Indexing Latent Memories for Access-Aware Continual Instruction Tuning of LLMs](<https://devfeed.tech/articles/self-indexing-latent-memories-for-access-aware-continual-instruction-tuning-of-llms-126155.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11684>)

Author: Dhruv Dixit, Paritosh Pandey, Snehalkumar 'Neil' S. Gaikwad

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [acquisition](<https://devfeed.tech/tags/acquisition.md>), [adapter](<https://devfeed.tech/tags/adapter.md>), [bank](<https://devfeed.tech/tags/bank.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [gathering](<https://devfeed.tech/tags/gathering.md>), [indexing](<https://devfeed.tech/tags/indexing.md>), [inference](<https://devfeed.tech/tags/inference.md>), [information](<https://devfeed.tech/tags/information.md>), [keys](<https://devfeed.tech/tags/keys.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [module](<https://devfeed.tech/tags/module.md>), [modules](<https://devfeed.tech/tags/modules.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [order](<https://devfeed.tech/tags/order.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [points](<https://devfeed.tech/tags/points.md>), [private](<https://devfeed.tech/tags/private.md>), [prompt](<https://devfeed.tech/tags/prompt.md>)

### AI overview

TILA-CT addresses continual instruction tuning's module-selection problem by giving each task a latent memory and adapter bank in a frozen LLM. The memory gathers information from the causal prompt prefix, injects task-conditioned context during adaptation, and supplies a persistent routing prototype to a shared prompt-only router at inference. On an 8-task CT-Pipeline stream, it reports 65.19 acquisition and 63.98 final macro scores with 1.38 points of average forgetting; a middle-reversed order yields 65.38 and 64.17. Across six backbones, final scores stay within 1.5 points of acquisition. Reported routing accuracy is 93.9%, while oracle routing recovers 1.21 final macro points, indicating some remaining gap in access to preserved task computation.

### Source excerpt

arXiv:2611.11684v1 Announce Type: new Abstract: Continual instruction tuning often preserves old skills with task-private modules. Inference still requires selecting the right module from the prompt alone. We introduce Task-Isolated Latent Adapter Continual Training (TILA-CT), a replay-free method that makes each task module self-indexing. Each task receives a latent memory and adapter bank in a frozen LLM. During adaptation, the memory gathers causal-prefix information and injects task-conditioned context. For access, its persistent state provides the routing prototype used by a shared prompt-only router. On the 8-task CT-Pipeline stream, TILA-CT achieves 65.19 acquisition and 63.98 final macro score, with 1.38 points of average forgetting. A middle-reversed order gives 65.38 acquisition and 64.17 final macro score. Across 6 backbones, final scores remain within 1.5 points of acquisition. Ablations support memory-derived routing keys and causal gathering. Final macro routing accuracy is 93.9\%, and oracle routing recovers 1.21 final macro points. These results separate preserved task computation from learned access to it.

## Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

DevFeed: [Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference](<https://devfeed.tech/articles/ensemble-routing-for-prompt-adaptive-cost-aware-llm-inference-125896.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10193>)

Author: Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [capability](<https://devfeed.tech/tags/capability.md>), [cardinality](<https://devfeed.tech/tags/cardinality.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [composition](<https://devfeed.tech/tags/composition.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [models](<https://devfeed.tech/tags/models.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

PISA estimates individual model capabilities from soft cluster memberships and model-specific predictors to route prompts to a single model. PISA-E extends this approach by selecting both ensemble composition and size according to an inference-time cost-quality preference, without retraining or ensemble-specific training. Across four routing benchmarks, PISA had the highest macro-average cost-performance hypervolume among evaluated single-model routers in both in-distribution and out-of-distribution settings. When prioritizing quality, PISA-E had the highest macro-average task performance among evaluated ensemble approaches, exceeding the strongest competitors by 2.09 and 2.87 percentage points, respectively. At selected lower-cost operating points, it also exceeded those competitors' task performance while reducing average inference cost by approximately 36% and 25%, respectively.

### Source excerpt

arXiv:2611.10193v1 Announce Type: new Abstract: Large language models (LLMs) increasingly support Web-based services that must balance response quality and inference cost across heterogeneous requests. Conventional routing selects one model per prompt, while fixed ensembles cannot adapt their composition and cost to individual requests. We introduce PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. At the core of both methods, PISA combines soft cluster memberships with model-specific predictors to estimate individual model capabilities. Building on these estimates, PISA-E jointly selects ensemble composition and cardinality under an explicit cost--quality preference that can be adjusted at inference time without retraining. A tractable ensemble-success surrogate and efficient candidate selection enable this extension without ensemble-specific training. Across four LLM routing benchmarks under in-distribution (ID) and out-of-distribution (OOD) conditions, PISA achieves the highest macro-average cost--performance hypervolume among the evaluated single-model routers in both settings. Meanwhile, PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches when prioritizing response quality, surpassing the strongest competitors by 2.09 and 2.87 percentage points under ID and OOD conditions, respectively. At selected lower-cost operating points, it also exceeds the strongest ensemble competitors' macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively. These results demonstrate the effectiveness of prompt-adaptive capability estimation for both single-model routing and selective ensembling.

## Dynamic model routing: The new SD-WAN for the AI era

DevFeed: [Dynamic model routing: The new SD-WAN for the AI era](<https://devfeed.tech/articles/dynamic-model-routing-the-new-sd-wan-for-the-ai-era-124198.md>)

Original publisher: [Read original article](<https://blogs.cisco.com/networking/dynamic-model-routing-the-new-sd-wan-for-the-ai-era>)

Author: Murtaza Doctor

Published: 2026-11-09T20:00:59Z

Content type: article

Language: en

Sources: [Cisco Blogs](<https://devfeed.tech/sources/cisco-blogs.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [SD-WAN](<https://devfeed.tech/topics/sd-wan.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agenticops](<https://devfeed.tech/tags/agenticops.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [air-gapped](<https://devfeed.tech/tags/air-gapped.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [cisco-deep-network-model](<https://devfeed.tech/tags/cisco-deep-network-model.md>), [cost](<https://devfeed.tech/tags/cost.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [model-routing](<https://devfeed.tech/tags/model-routing.md>), [networking](<https://devfeed.tech/tags/networking.md>), [performance](<https://devfeed.tech/tags/performance.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sd-wan](<https://devfeed.tech/tags/sd-wan.md>), [security](<https://devfeed.tech/tags/security.md>), [token](<https://devfeed.tech/tags/token.md>), [token-efficiency](<https://devfeed.tech/tags/token-efficiency.md>), [wan](<https://devfeed.tech/tags/wan.md>)

### AI overview

Dynamic model routing evaluates each AI request against complexity, cost, latency, privacy, and governance requirements, then directs it to an appropriate endpoint such as a deterministic API, specialized small model, private model, or frontier LLM. The article argues that enterprise routing should add assurance by using network, cloud, and security telemetry to validate end-to-end performance and policy before processing. Its proposed ensemble approach runs routine monitoring and checks without model calls, uses domain-specific models for bounded tasks, and escalates to deeper reasoning when needed. Local or private deployments can reduce latency and keep sensitive data on premises; automated checkpoints can also limit runaway agent spending. The article cites up to 30x lower operational costs for specialized engines on some tasks, while emphasizing cost per verified outcome over raw token reduction.

### Source excerpt

Stop overspending on monolithic LLMs. Learn how dynamic model routing can intelligently steer queries based on performance, cost, and security requirements to help maximize your AI ROI and optimize token efficiency.

## CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation

DevFeed: [CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation](<https://devfeed.tech/articles/cascaderec-deployed-adaptive-llm-compute-allocation-for-multi-tenant-content-recommendation-123281.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08566>)

Author: Kai Zhao, Guilin Zhang, Xiang Cheng, Xiquan Cui, Henry Heng, Xu Chu, Alet Blanken

Published: 2026-11-09T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-eks](<https://devfeed.tech/tags/aws-eks.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [compute](<https://devfeed.tech/tags/compute.md>), [compute-allocation](<https://devfeed.tech/tags/compute-allocation.md>), [content](<https://devfeed.tech/tags/content.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [gke](<https://devfeed.tech/tags/gke.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>)

### AI overview

CascadeRec uses a lightweight Tier-1 model for every recommendation request and routes selected requests to more expensive Tier-2 inference, such as self-consistency sampling, multi-path reasoning, or a larger model. In a production A/B experiment at an FP&A SaaS provider, this approach reduced inference cost by about 10% versus uniform use of the expensive tier, with minimal latency impact and no quality regression observed to date. Public benchmark analysis found that uniform LLM reranking lowered NDCG@10 by up to 30% for users with sparse histories; its only statistically significant gain was for users with the longest histories, and escalating cold users could worsen losses. Because beneficial segments vary by domain, the production rollout gates escalation by segment. The abstract also identifies tenant load imbalance and serving-layer handling of output parsing failures as deployment lessons.

### Source excerpt

arXiv:2611.08566v1 Announce Type: cross Abstract: Large language models (LLMs) are attractive rerankers for recommendation. Running an expensive model on every request is costly, and for users with sparse interaction histories it often lowers quality. We present CascadeRec, a two-tier LLM reranking cascade deployed in production at a large financial planning and analysis (FP&A) SaaS provider. It serves content recommendations to enterprise tenants on AWS EKS and GKE. A lightweight Tier-1 model reranks every request. A router escalates only the requests where extra inference-time compute is likely to help, and sends them to a more expensive Tier-2 procedure (self-consistency sampling, multi-path reasoning, or a larger model). The system has run in a production A/B experiment for over a month. Compared with uniform use of the expensive tier, CascadeRec has reduced inference cost by about 10%, with minimal latency impact and no quality regression observed to date. Tenant data cannot be released, so we also evaluate the pipeline on public benchmarks (MovieLens-1M and Amazon-Electronics). Segment-level analysis with paired-bootstrap confidence intervals shows that uniform LLM reranking lowers NDCG@10 for sparse-history users by up to 30%, and its only statistically significant gain appears on the segment with the longest histories. Escalating cold users can make the loss larger instead of repairing it. Which segments benefit from escalation depends on the domain, and this motivates the segment-gated rollout we use in production. We describe the deployment architecture, the measured benefits, and lessons learned, including inter-tenant load imbalance as a driver for cascading and the handling of LLM output-parsing failures at the serving layer.

## Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

DevFeed: [Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill](<https://devfeed.tech/articles/scout-before-you-route-predicting-task-success-and-token-count-from-a-small-llm-prefill-123343.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09028>)

Author: Aristides Milios, Christopher Pal

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [few-shot-learning](<https://devfeed.tech/topics/few-shot-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [route](<https://devfeed.tech/tags/route.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

A small frozen LLM can use prefill activations to predict candidate models' success probabilities and output lengths, enabling per-query cost estimates for routing. With reasoning models, perfect length knowledge could cut spending 29-61% on eight of nine datasets; the learned predictions save 9-33% at matched accuracy versus median-length pricing, reaching up to 69% of the ideal savings. Length is largely problem-dependent: on most math and code datasets, harder problems tend to produce longer outputs, so success predictions alone approximate length-based pricing; this breaks down on knowledge and mixed-format data. A length probe retains most savings with 20 labelled examples and transfers without labels to four of seven new benchmarks. Savings plateau around a 2B scout, while scouts above 4B cost more than the additional savings justify. These results are reported for the evaluated datasets and models.

### Source excerpt

arXiv:2611.09028v1 Announce Type: cross Abstract: Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.

## When LLM Routers Overpay: The Cost of Strong-Model Preference

DevFeed: [When LLM Routers Overpay: The Cost of Strong-Model Preference](<https://devfeed.tech/articles/when-llm-routers-overpay-the-cost-of-strong-model-preference-119995.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.03478>)

Author: Guannan Lai, Long Chen, Han-Jia Ye

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [routers](<https://devfeed.tech/tags/routers.md>), [utility](<https://devfeed.tech/tags/utility.md>)

### AI overview

Existing LLM routers may over-select the strongest, most expensive model as cost budgets loosen, even when cheaper feasible models offer comparable or identical utility. The abstract attributes this partly to persistent model-level preference for costly models, which budget relaxation exposes; ambiguity alone does not explain the overpayment. EquiRouter learns query-dependent model rankings, preserves meaningful utility differences, and favors lower-cost options among equivalent candidates using a Cost-Aware Ranking Objective and a lightweight Query-Model Interaction Representation. The authors report improved cost efficiency with competitive peak and whole-curve performance across text, multimodal, continuous-utility, and large-model-pool settings.

### Source excerpt

arXiv:2602.03478v2 Announce Type: replace Abstract: LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed. However, we find that existing routers often violate this principle under loose cost budgets: as the budget increases, they increasingly select the strongest and most expensive model even when less expensive feasible candidates provide comparable or identical utility. We refer to this behavior as strong-model over-selection under loose budgets. Our analysis shows that ambiguity alone does not explain this phenomenon. Substantial overpayment can arise when differently priced models offer equivalent or near-equivalent utility and the router retains a persistent model-level preference for costly models; budget relaxation exposes this preference by making premium candidates feasible. Motivated by this diagnosis, we propose EquiRouter, a decision-aligned router that learns query-dependent model rankings, preserves meaningful utility differences, and explicitly prefers lower-cost models among equivalent candidates through a Cost-Aware Ranking Objective and a lightweight Query--Model Interaction Representation. Experiments across diverse text, multimodal, continuous-utility, and large-model-pool settings show that EquiRouter consistently improves cost efficiency while maintaining competitive peak and whole-curve performance. Code is available at https://github.com/LAMDA-Model-Reuse/EquiRouter.

## Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents

DevFeed: [Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents](<https://devfeed.tech/articles/hard-issue-weak-model-or-bad-run-rethinking-failure-prediction-for-coding-agents-120836.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08285>)

Author: Xingyue Shi, Wesley K. G. Assun\\c{c}\\~ao

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Model Routing](<https://devfeed.tech/topics/model-routing.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [failure](<https://devfeed.tech/tags/failure.md>), [issue](<https://devfeed.tech/tags/issue.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [run](<https://devfeed.tech/tags/run.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [switching](<https://devfeed.tech/tags/switching.md>)

### AI overview

On public SWE-bench Verified trajectories, failure predictors mostly identify issue difficulty and model ability rather than whether a particular run is going wrong: an issue-difficulty reference reaches 0.92 AUROC, compared with about 0.62 for trajectory predictors, whose added signal beyond difficulty and ability is statistically equivalent to zero. Per-issue model switching gains nothing on held-out runs. A repeated-run pilot finds the same model can both solve and fail an issue, so retries can address bad-run failures but not issues beyond the model's ability. The authors call for predictors that explain failure causes, evaluated on repeated-run benchmarks and in closed-loop, equal-budget settings.

### Source excerpt

arXiv:2611.08285v1 Announce Type: new Abstract: Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong. If it is going wrong, the run can be stopped, retried, or switched to another model. We ask whether these predictors learn anything about the run itself. On public SWE-bench Verified trajectories, where each model attempts each issue once, two run-independent factors dominate predictor performance. The first is issue difficulty: a reference score that never looks at the run, only at how hard the issue is for other models, reaches 0.92 AUROC, whereas trajectory predictors reach about 0.62. The second is model ability: within an issue, knowing which model ran predicts failure better than our predictors do, and the predictors largely reproduce the overall ability ranking of the models. Beyond difficulty and ability, what predictors gain from the trajectory is statistically equivalent to zero. This confusion carries over to the actions taken after a prediction. Model switching is often motivated by the gap between the best single model's resolution rate and an any-model oracle. This gap is no larger than random variation alone produces. Per-issue switching gains nothing on held-out runs. In a repeated-run pilot, the same model often solves an issue in some runs and fails it in others. Retrying thus has real room to repair failures, but it rescues only failures caused by a bad run, not those on an issue beyond the model's ability. Current predictors cannot tell the causes apart, as they mostly predict the issue and the model, not the run. We call for failure prediction that explains why a run fails, evaluated on repeated-run benchmarks and in closed-loop, equal-budget settings.

## Introducing the New Auto and Auto Code Routers

DevFeed: [Introducing the New Auto and Auto Code Routers](<https://devfeed.tech/articles/introducing-the-new-auto-and-auto-code-routers-122068.md>)

Original publisher: [Read original article](<https://openrouter.ai/blog/announcements/auto-and-auto-code-routers/>)

Author: Brian Thomas

Published: 2026-11-06T00:00:00Z

Content type: release

Language: en

Sources: [OpenRouter](<https://devfeed.tech/sources/openrouter.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [ai-governance](<https://devfeed.tech/topics/ai-governance.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [effort](<https://devfeed.tech/tags/effort.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [generation](<https://devfeed.tech/tags/generation.md>), [governance](<https://devfeed.tech/tags/governance.md>), [health](<https://devfeed.tech/tags/health.md>), [live](<https://devfeed.tech/tags/live.md>), [model](<https://devfeed.tech/tags/model.md>), [next](<https://devfeed.tech/tags/next.md>), [policies](<https://devfeed.tech/tags/policies.md>), [pool](<https://devfeed.tech/tags/pool.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [request](<https://devfeed.tech/tags/request.md>), [router](<https://devfeed.tech/tags/router.md>), [routers](<https://devfeed.tech/tags/routers.md>), [state](<https://devfeed.tech/tags/state.md>)

### AI overview

The new Auto Router and coding-focused Auto Code select a model and reasoning effort from the models allowed by governance policies, weighing eleven request factors alongside live prices, provider health, benchmark performance, and cache state. Auto Code favors a higher tier of coding models, prioritizes latency, and responds more strongly to failures; Auto Router targets mixed traffic and cost efficiency. The router can adjust reasoning effort before switching models to preserve cached input, and session results inform later decisions. The article reports Auto Code benchmark scores in line with Claude Opus 5.5 at 74% lower cost per task, and says the new router was up to 4x better on benchmarks and 3x faster at p90 in a live routing test. Routers can trade some peak performance for consistent cost and quality; the article notes no router beats the best frontier model at every point on the cost-quality curve.

### Source excerpt

The next generation of Auto Router, rebuilt on a new technical foundation, plus Auto Code, a new router specialized for coding work. Both weigh eleven factors per request against live pricing, provider health, benchmarks, and cache state to pick a model and reasoning effort from the model pool your governance policies allow.

[Next page](<https://devfeed.tech/topics/model-routing.md?cursor=WyIyMDI2LTExLTA2VDAwOjAwOjAwKzAwOjAwIiwgImU4M2E2ODljLTM5YjUtNGMyMC1hMjIzLWUxZmE4Y2RlY2EwYyJd>)