# Model Routing Technique for selecting a large language model from a candidate pool for each request based on its requirements. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## GitHub Copilot's Project HydraFusion Promises Frontier Level Performance Through Multi-Model Routing DevFeed: [GitHub Copilot's Project HydraFusion Promises Frontier Level Performance Through Multi-Model Routing]() Original publisher: [Read original article]() Author: Olimpiu Pop Published: 2026-09-13T06:06:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [GitHub Copilot](), [Model Routing](), [AI-assisted coding](), [Orchestration]() Tags: [agentic-coding](), [ai](), [ai-ml-data-engineering](), [benchmarks](), [development](), [frontier-model](), [github-copilot](), [github-hydrafusion](), [ml-data-engineering](), [model-routing](), [news](), [orchestration]() ### AI overview GitHub's Project HydraFusion research preview for Copilot orchestrates multiple models at runtime for coding tasks. Its single, cascade, and critique execution patterns aim to balance task quality, latency, and estimated cost. ### Source excerpt GitHub's Project HydraFusion is a research preview for GitHub Copilot that enhances coding intelligence through runtime model orchestration. It dynamically assembles execution plans using models from various providers. The system employs three execution patterns based on task complexity. Evaluations indicate that it achieves high task quality while significantly reducing operational costs. By Olimpiu Pop ## Model routing with Google Cloud API Gateway DevFeed: [Model routing with Google Cloud API Gateway]() Original publisher: [Read original article]() Author: Mak Ahmad; Sanjay Pujare Published: 2026-09-12T11:04:33.891311Z Content type: article Language: en Sources: [Google Developers Blog]() Topics: [Amazon API Gateway](), [Model Routing](), [Google](), [Cloud](), [OpenAPI Specification](), [Serverless](), [Claude](), [OpenAI]() Tags: [ai-gateway](), [api-gateway](), [claude](), [cloud](), [google](), [model-routing](), [models](), [openai](), [openapi](), [serverless]() ### AI overview Google Cloud API Gateway adds model routing in Public Preview, providing a serverless, OpenAI-compatible endpoint that dynamically routes requests to Gemini, Claude, or OpenAI OSS-GPT. Developers can configure routing in OpenAPI specifications, centralize model changes, and separate application authentication from backend model credentials. ### Source excerpt Google Cloud API Gateway now offers a model routing feature in Public Preview, allowing developers to dynamically route traffic to models like Gemini, Claude, or OpenAI OSS-GPT without hardcoding endpoints or managing open-source proxies. Developers can easily configure these routing rules directly within their OpenAPI 3.x specifications by mapping virtual model names to specific backend targets on a shared host. Once deployed, the Gateway acts as a serverless ingress layer that accepts standard OpenAI-compatible requests, automatically transcodes the payload to the native schema of the target model, and routes the traffic on the fly. ## How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing DevFeed: [How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing]() Original publisher: [Read original article]() Author: Ivan Klaric Published: 2026-09-10T17:22:31Z Content type: article Language: en Sources: [The Pragmatic Engineer]() Topics: [Model Routing](), [Artificial Intelligence](), [Inference](), [LLM evaluation / benchmarking](), [Benchmark]() Tags: [ai](), [benchmark](), [benchmarking](), [cost](), [inference](), [model-routing](), [models](), [optimize]() ### AI overview The article reports that companies including Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are reducing AI costs by using open models, model routing, benchmarking, and other optimization techniques. It details Uber's use of cheaper inference services, optimized model selection, smaller subagent models, prompt caching, and automatic compaction. ### Source excerpt Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are making large savings on their AI bills by dropping proprietary models and using smart model routing. ## How Smart Model Routing Can Cut LLM Costs by 10X DevFeed: [How Smart Model Routing Can Cut LLM Costs by 10X]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-09-09T15:30:26Z Content type: article Language: en Sources: [ByteByteGo]() Topics: [Model Routing](), [Large Language Model](), [API]() Tags: [cost](), [llm](), [model-routing](), [models](), [routing](), [routing-system](), [tokens]() ### AI overview This article explains how smart model routing can reduce LLM application costs by sending simple requests to less expensive models and difficult requests to more capable models. It notes that savings and response quality depend on request types, model price differences, and routing performance, and discusses router, cascading, semantic, and learned routing approaches. ### Source excerpt Cost reduction isn't a given. It also depends on the types of requests the application receives, the price difference between models, and how well the routing system performs. In this article, we are going to look at various aspects ## LLM Routing Can Cost More Than Not Routing DevFeed: [LLM Routing Can Cost More Than Not Routing]() Original publisher: [Read original article]() Author: Avi Chawla Published: 2026-09-07T20:30:58Z Content type: article Language: en Sources: [Daily Dose of Data Science]() Topics: [Model Routing](), [Large Language Model](), [Frontier Model](), [coding]() Tags: [agent](), [cost](), [llm](), [model](), [models](), [routing]() ### AI overview The article examines why routing requests among LLMs can cost more than using one model throughout an agent session. Although cheaper models may handle simpler tasks, routing adds overhead and can be counterproductive inside agent loops. ### Source excerpt ...covered with a production-grade router for LLM apps. ## Project HydraFusion: Frontier quality via multi-model orchestration DevFeed: [Project HydraFusion: Frontier quality via multi-model orchestration]() Original publisher: [Read original article]() Author: GitHub Staff Published: 2026-09-04T16:04:14Z Content type: release Language: en Sources: [GitHub Engineering]() Topics: [Orchestration](), [Model Routing](), [GitHub Copilot](), [AI-assisted coding](), [Optimization](), [Claude]() Tags: [agentic-coding](), [ai-ml](), [benchmarking](), [benchmarks](), [claude](), [cost](), [github-copilot](), [llms](), [optimization](), [orchestration](), [routing]() ### AI overview GitHub introduces Project HydraFusion, a GitHub Copilot research preview that selects multi-model execution workflows to balance quality, cost, and latency for coding tasks. ### Source excerpt In controlled offline evaluations, HydraFusion's selective coding workflows matched or exceeded the evaluated Opus 5 baseline while reducing estimated workflow cost. Now available as a research preview in GitHub Copilot. The post Project HydraFusion: Frontier quality via multi-model orchestration appeared first on The GitHub Blog. ## A Model Portfolio for cost-efficient AI across the software development lifecycle DevFeed: [A Model Portfolio for cost-efficient AI across the software development lifecycle]() Original publisher: [Read original article]() Author: Praveen Sidda Published: 2026-09-01T07:16:01Z Content type: article Language: en Sources: [Data Science at Microsoft]() Topics: [AI Development](), [AI Models](), [Model Routing](), [sdlc](), [Agile](), [Developer experience](), [Compression](), [Requirements](), [Testing](), [Deployment](), [Code review]() Tags: [agentic-sdlc](), [ai](), [ai-cost-optimization](), [ai-models](), [code-review](), [compression](), [deployment](), [github-copilot](), [llm](), [overhead](), [requirements](), [software-development]() ### AI overview The article examines whether routing software development tasks across a portfolio of AI models can reduce costs compared with using one premium model. It reports that lower-cost models handled well-defined tasks, while premium models were reserved for architecture, implementation, and code review; context compression reduced token usage but risked losing important technical details. ### Source excerpt Image generated by AIWhat one controlled experiment taught me about matching model capability to developer work Topic: Can intelligently routing developer tasks across different AI models outperform relying on a single premium model? In this article, I put that question to the test by mapping software development lifecycle (SDLC) stages to a portfolio of AI models and comparing the outcomes. Motivation As AI becomes embedded throughout the AI-Native Development Lifecycle (AIDLC), an evolution of the traditional Software Development Lifecycle (SDLC), its cost is no longer tied to a single prompt. A single developer task can involve multiple model calls, each carrying source files, conversation history, tool definitions, and generated output. Applying the most capable model to every interaction is straightforward, but it also consumes premium model capacity on tasks that less expensive models can often complete just as effectively. This raises an important question for engineering organizations: How can teams reduce the cost of AI-assisted development without compromising quality, reliability, or the developer experience? My first instinct was to reduce token consumption. Context compression appeared to be the most direct path to lowering inference costs by shortening prompts. Although it reduced token usage, it also introduced risk. Important constraints and technical details could be lost, affecting downstream tasks. Source code, stack traces, and active instructions proved to be especially poor candidates for lossy compression. That experience shifted my focus. The objective was not to process fewer tokens, but to complete developer tasks successfully at a lower overall cost. I then experimented with model allocation. Lower-cost models handled well-defined tasks such as requirements synthesis, planning, routine test generation, deployment artifacts, and final summaries, while premium models were reserved for architecture, implementation, and code review. This appro ## The builder's guide to GPT-5.6 DevFeed: [The builder's guide to GPT-5.6]() Original publisher: [Read original article]() Published: 2026-08-13T11:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [Artificial Intelligence](), [Model Routing](), [API](), [Orchestration](), [Benchmark]() Tags: [agentic](), [agents](), [ai](), [ai-agents](), [api](), [applied-ai](), [benchmark](), [build-faster](), [compute](), [cost](), [efficiency](), [frontier-model](), [gpt](), [latency](), [model](), [performance](), [responses](), [search](), [tokens](), [workflows]() ### AI overview A technical guide to using GPT-5.6 in production AI agents. It covers smarter model selection, cost-efficient reasoning, Responses API controls, multi-agent orchestration, programmatic tool calling, and the use of smaller models for high-volume or latency-sensitive workflows. ### Source excerpt Learn how startups use GPT-5.6 to build faster, more cost-efficient AI agents with smarter model selection and new Responses API capabilities. ## Making Knowledge Distillation Cheap Enough to Run at Scale DevFeed: [Making Knowledge Distillation Cheap Enough to Run at Scale]() Original publisher: [Read original article]() Author: Antonio Tiene; Iker García-Ferrero; Ali Hashemi; Bakbergen Ryskulov Published: 2026-08-10T10:05:36Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [LLM Techniques](), [Large Language Model](), [Post-training optimization](), [Model Routing](), [GPU](), [Machine learning](), [PyTorch](), [gpt-oss](), [Nvidia]() Tags: [blog](), [caching](), [compute](), [cost](), [gpt](), [gpt-oss](), [gpu](), [hugging-face](), [language-models](), [large-language-models](), [long-context](), [machine-learning](), [memory](), [performance](), [pytorch](), [qwen](), [research](), [training]() ### AI overview This article presents an efficient approach to knowledge distillation for large language models. It caches the teacher model's top-K logits offline and uses a fused, memory-efficient chunked KL-divergence loss, avoiding simultaneous teacher-student residency and full vocabulary-by-sequence-length matrices. The changes reduce VRAM use and training cost, enabling long-context distillation on a single GPU and making larger-scale experimentation more practical. ### Source excerpt A Blog post by Multiverse Computing on Hugging Face ## The AI Gateway Buyer's Guide: Beyond Routing and Tool Visibility DevFeed: [The AI Gateway Buyer's Guide: Beyond Routing and Tool Visibility]() Original publisher: [Read original article]() Author: Ritesh Patel Published: 2026-08-02T16:58:40Z Content type: opinion Language: en Sources: [Nirmata]() Topics: [Artificial Intelligence](), [Model Routing](), [ai security](), [agent observability](), [Model Context Protocol]() Tags: [ai](), [ai-gateway](), [ai-security](), [claude](), [cost-optimization](), [engineering](), [kubernetes](), [latency](), [mcp](), [routing]() ### AI overview This opinion article argues that AI gateways should not be treated as governance systems merely because they provide model routing and tool-call visibility. Routing can optimize cost and latency, while monitoring can show which tools or MCP servers were used, but governance requires deciding whether an agent action is permitted for a specific agent, with specific arguments, at a specific time. ### Source excerpt Over the past year, nearly every engineering org I talk to has reached the same milestone: AI agents are no longer a demo. They're calling real tools, against real systems, with real consequences. And nearly every one of those orgs has reached for the same... The post The AI Gateway Buyer's Guide: Beyond Routing and Tool Visibility first appeared on Nirmata. ## Harness AI Configs for Runtime Controls of AI Behavior DevFeed: [Harness AI Configs for Runtime Controls of AI Behavior]() Original publisher: [Read original article]() Author: Nico Zelaya Published: 2026-07-21T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [Artificial Intelligence](), [configuration](), [feature flags](), [experiments](), [Model Routing](), [Inference]() Tags: [ai](), [blog](), [config](), [experimentation](), [inference](), [policy](), [routing]() ### AI overview Harness AI Config Management provides a governed runtime configuration layer for changing prompts, models, routing, inference parameters, and other AI behavior without redeploying code. It supports targeting, experimentation, approvals, policy controls, versioning, and audit trails. ### Source excerpt Harness AI Config Management helps teams change prompts, models, and AI behavior at runtime with targeting, experimentation, approvals, policy, and audit trails | Blog ## Model Routing Is Simple. Until It Isn't. DevFeed: [Model Routing Is Simple. Until It Isn't.]() Original publisher: [Read original article]() Author: Yara Rizk; Eyal Shnarch; Jason Tsay; Merve Unuvar Published: 2026-07-15T17:27:01Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Model Routing](), [Optimization](), [systems](), [Caching](), [Claude](), [anthropic]() Tags: [agentic](), [caching](), [claude](), [compliance](), [cost](), [enterprise](), [latency](), [model-routing](), [optimization](), [privacy](), [production](), [reasoning](), [systems](), [tool]() ### AI overview Model routing in agentic systems is more than classifying task difficulty or comparing list prices. Effective cost depends on the interaction between the model, workload, and serving infrastructure, including cache-hit rates and cache-read pricing. Production routers must balance cost, quality, latency, specialization, reliability, and enterprise governance constraints such as compliance, data residency, privacy, and approved-model policies. ### Source excerpt Except it's not. Most routing systems assume that model selection is a classification problem. In our experience building routing into agentic systems, what looks like a model-selection problem quickly becomes a systems optimization problem. Three dimensions made this surprisingly hard for us. We expected GPT-4.1 to be cheaper than Claude Sonnet 4.6. It wasn't. ## The Pulse: a new trend, smart model routing DevFeed: [The Pulse: a new trend, smart model routing]() Original publisher: [Read original article]() Author: Gergely Orosz Published: 2026-07-02T18:46:24Z Content type: article Language: en Sources: [The Pragmatic Engineer]() Topics: [Model Routing](), [AI Models](), [LLMs]() Tags: [ai-gateway](), [ai-models](), [latency](), [litellm](), [model](), [model-routing](), [openrouter]() ### AI overview The article surveys emerging smart model-routing tools that select AI models for coding and other tasks based on factors such as prompt difficulty, cost, latency, and availability. It lists offerings including Factory Router, Not Diamond, Vercel AI Gateway, OpenRouter, Kilo Gateway, Requestly.ai, LiteLLM, and others, while noting that some vendors claim cost savings. ### Source excerpt Are there any 'intelligent' router solutions out there which select the right model for the right task? I looked into it, and there are a few options. ## LLM router architecture: best practices for 2026 DevFeed: [LLM router architecture: best practices for 2026]() Original publisher: [Read original article]() Author: Jim Allen Wallace Published: 2026-07-01T00:00:00Z Content type: tutorial Language: en Sources: [Redis Blog]() Topics: [Model Routing](), [LLMs](), [Latency](), [Availability](), [API](), [networking]() Tags: [architecture](), [availability](), [best-practices](), [cost](), [latency](), [llm](), [outage](), [router](), [routing](), [tech-de]() ### AI overview This guide explains model routers as middleware that selects the most suitable large language model for each request based on factors such as task complexity, cost, latency, availability, load, and safety requirements. It covers common routing strategies, production architecture patterns, proxy and recommendation-based designs, and how routing can reduce costs and improve reliability through fallback handling. ### Source excerpt You picked GPT-5 for every LLM call in your app because it was the safe call: chat, autocomplete, classification, summarization, all of it. Then the bill arrived, and you traced part of it back to queries like "what are your business hours?" getting r... ## How Okara runs CMO agents for 120,000 companies on Vercel DevFeed: [How Okara runs CMO agents for 120,000 companies on Vercel]() Original publisher: [Read original article]() Author: Eric Dodds Published: 2026-06-11T04:00:00Z Content type: article Language: en Sources: [Vercel News]() Topics: [Vercel](), [AI Infrastructure](), [Model Routing](), [API](), [SDKs](), [Back end]() Tags: [agents](), [ai-gateway](), [ai-infrastructure](), [api](), [backend](), [open-source](), [routing](), [sdks](), [vercel]() ### AI overview Okara uses Vercel and Vercel AI Gateway to run an AI CMO that coordinates specialized marketing sub-agents for more than 120,000 businesses. AI Gateway consolidates eight model-provider integrations behind one API key and endpoint, handling retries, fallback routing, provider monitoring, and zero-data retention support while Okara processes 4 billion tokens daily. ### Source excerpt Okara on Vercel 4 billion tokens processed daily across a multi-provider AI stack on Vercel AI CMOs actively managing growth for 120,000+ businesses Eight sub-agents handling SEO, GEO, social, content, Reddit, and Hacker News New AI models available to users the same day they ship Okara is an AI CMO that directs a team of specialized sub-agents to drive marketing, so founders don't have to. Give Okara your website URL, and the AI CMO builds a marketing strategy, develops a brand voice, and activates agents across SEO, content, and social media to drive awareness and pipeline without a single marketing hire. As Fatima Rizwan, Okara's founder, puts it: "You can build something in a weekend and spend months trying to get anyone to notice." Distribution, she argues, is stuck in the pre-AI era: fragmented across subscriptions and agencies that cost over $15,000 a month before a single dollar comes back. Okara is built by a team of four and processes 4 billion tokens a day. The company operates on the new model for startups: a small team building a platform that handles growth for thousands of other companies. Fatima quickly learned that serving hundreds of thousands of companies with four people meant infrastructure had to be invisible. Any time spent on it was time not spent building. Using AI Gateway to integrate multiple providers with one API keyThe friction of managing individual provider SDKs Okara's backend originally talked to eight model providers through separate SDKs, each with its own key management, image handling, and edge cases. When they expanded to open-source models, the approach broke down completely. Every new model meant an engineer stopped shipping product and wrote an adapter instead. Retry logic, fallback routing, and provider health monitoring all lived in Okara's codebase, maintained by hand. Most AI infrastructure would have required Okara to keep living with that friction. That's why they moved to Vercel AI Gateway. One endpoint, every provide ## OpenCode Now Supports DigitalOcean Inference Router for Intelligent Model Routing DevFeed: [OpenCode Now Supports DigitalOcean Inference Router for Intelligent Model Routing]() Original publisher: [Read original article]() Author: Musa Malik Published: 2026-05-28T21:02:42Z Content type: release Language: en Sources: [DigitalOcean]() Topics: [Digital Ocean](), [Model Routing](), [AI Inference](), [AI-assisted coding](), [Open Source](), [coding]() Tags: [ai-coding](), [ai-inference](), [api](), [cost](), [digitalocean](), [latency](), [model-routing](), [open-source](), [product-updates]() ### AI overview DigitalOcean's Inference Router is in public preview and can be accessed through OpenCode, an open-source AI coding agent. It dynamically routes requests across models to help developers manage latency, cost, and output quality. ### Source excerpt Coding agents today have a massive spending problem. Every request, whether you're designing system architecture or writing a single-line docstring, often gets routed to the same expensive frontier model. The result: unnecessary token usage, higher inference costs, and little awareness of task complexity or budget constraints. This high cost stems from a "one-size-fits-all" approach to model usage, where premium frontier models are utilized for trivial tasks that don't require such intensive reasoning effort. In multi-agent workflows, where orchestrators delegate work to specialized subagents, this lack of discrimination frequently leads to runaway costs and opaque failure modes. Without intelligent routing, developers can essentially be forced into closed-provider lock-in and high API usage fees, which quickly escalate during exploratory building phases. DigitalOcean Inference Router, now in Public Preview, was built to solve this problem by dynamically routing requests to the right model for the job. As part of DigitalOcean's AI-Native Cloud, it gives developers a unified way to control, optimize, and evaluate AI inference across models. And as of today, you can access it through OpenCode, the open-source AI coding agent, in as little as a few seconds. What is an Inference Router? An Inference Router is the auto-mode pattern engineers are used to, but with deliberate control over the tradeoffs that matter: latency, cost, and output quality. Rather than statically pointing your coding agent to a single model, an Inference Router can analyze each request and route it to the model best suited for that specific task. Not the most powerful model available, but the right model. That distinction is what drives real savings without compromising on your desired quality of output. To use DigitalOcean's Inference Router: Create an Inference Router from the router catalog--pick a preset or build a custom router via the API or UI. No GPU management, no infrastructure to run. Us ## How We Built DigitalOcean Inference Router DevFeed: [How We Built DigitalOcean Inference Router]() Original publisher: [Read original article]() Author: Adil Hafeez Published: 2026-05-20T14:57:13Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [Model Routing](), [Digital Ocean](), [Inference](), [Large Language Model](), [Multi Agent Systems]() Tags: [agentic-workflows](), [digitalocean](), [engineering](), [inference](), [llm](), [multi-agent-systems](), [product-updates](), [routing]() ### AI overview DigitalOcean describes its Inference Router, which uses purpose-built models and live metrics to route LLM requests to appropriate models. The article explains how routing can reduce the cost and maintenance burden of applying one frontier model uniformly across coding, analysis, debugging, and other agentic tasks. ### Source excerpt Most teams building on LLMs today make a single model decision and apply it uniformly across every request. They reach for a frontier model not because every task demands it, but because building the infrastructure to do anything smarter is hard, time-consuming, and easy to get wrong. When the tooling isn't there, the path of least resistance is to use a single model, even if it means that you end up overpaying for most tasks. Let's take an example. If you're a developer building with Cursor, Claude Code, Open Code or any coding agent today, you've already felt this. In a single session, your agent does deep codebase analysis, writes new functions, fixes bugs from test output, explains methods, searches documentation. These tasks are not equivalent but if you're on a single hardcoded model, you're paying frontier rates for all of them, including the ones that don't need it. The stakes are even higher in agentic workflows and multi-agent systems. When multiple agents are running in parallel each planning, executing, and evaluating across long-horizon tasks the cost of uniform model selection compounds with every step. Furthermore, major AI providers are moving toward token-based billing and tighter rate limits. Inference costs are about to get more expensive. The alternative is for hardcoded routing logic in the application layer with an intent classifier with the help of an LLM which adds to your cost and gets brittle fast. Even if you were to use a smaller model like Haiku to keep costs down, you're now paying for a routing call on top of every inference call. Also, accuracy takes a hit as the model is not purpose built for routing, and as your task types evolve or models change, the logic breaks in ways that are hard to catch. You've introduced double taxation: the cost of the classifier plus the cost of maintaining brittle routing code that needs updating with every change to your stack. You've turned model selection into a feature you own and maintain, which is ## Why AI Infrastructure Matters for Scaling LLM Workloads DevFeed: [Why AI Infrastructure Matters for Scaling LLM Workloads]() Original publisher: [Read original article]() Author: Amit Jotwani Published: 2026-05-13T16:45:02Z Content type: article Language: en Sources: [DigitalOcean]() Topics: [AI Infrastructure](), [Large Language Model](), [Model Routing](), [Serverless](), [GPU](), [API](), [Model Context Protocol (MCP)]() Tags: [ai](), [ai-infrastructure](), [api](), [digitalocean](), [engineering](), [gpu](), [infrastructure](), [llm](), [mcp](), [model-routing](), [serverless]() ### AI overview This DigitalOcean article argues that infrastructure around an LLM--including routing, data access, observability, and cost control--can matter more than model selection. It describes serverless inference, dedicated GPUs, and routing as stages that can scale on the same platform without re-platforming or rewriting code, and reports an example router reducing costs by about 80% across eight support tickets. ### Source excerpt Everyone calling an LLM API has access to the same models. So what actually sets technical teams apart? It's everything around the model like the routing logic, the live data pipelines, and the ability to scale from prototype to production without ever rewriting your code. Which LLM tops a benchmark matters less than what becomes possible when infrastructure stops being an afterthought, when one platform owns the full stack from GPU to API. Our Deploy 2026 session walked through this with live demos: serverless inference with web search and MCP tools added in a few lines, a break-even calculator for serverless versus dedicated, and a router built in the console in two minutes that cut costs by ~80% across a batch of eight support tickets. Moving between serverless, dedicated, and routed setups didn't require re-platforming, rewriting code, or switching providers, which is where most inference setups leave money on the table. Watch the full talk below, or keep reading for the rundown. View YouTube video Key takeaways from the Deploy Session: Models are increasingly a commodity. What sets teams apart now is everything around the model: routing, data access, observability, and cost control. Choosing a model is no longer the hard part. The infrastructure you start with should scale with you. The workflow should not force you to re-platform, renegotiate contracts, or rewrite code at every growth milestone. DigitalOcean owns its GPUs, networks, and data centers. As efficiency improves, your cloud bill decreases. As the hardware gets better, your workloads run faster. Your AI journey in three stages Most AI workloads move through three stages. You start small and serverless, you grow into dedicated GPUs when volume justifies it, and somewhere along the way, you want a router making per-request decisions for you. On most platforms, each of those stages means a new product, a new API, and a new contract. On DigitalOcean, there are three configurations of the same platform. T ## Predictive Human Preference: From Model Ranking to Model Routing DevFeed: [Predictive Human Preference: From Model Ranking to Model Routing]() Original publisher: [Read original article]() Author: Chip Huyen Published: 2024-02-28T00:00:00Z Content type: article Language: en Sources: [Chip Huyen]() Topics: [Model Routing](), [AI Models](), [Model Development](), [post-training](), [AI Chat]() Tags: [ai-models](), [anthropic](), [chatgpt](), [dpo](), [gemini](), [latency](), [model](), [model-routing](), [ranking](), [rlhf]() ### AI overview The article examines predictive human preference: predicting which AI model users will prefer for a specific prompt. It describes model routing as a use case, where prompts could be directed to a preferred model to potentially reduce cost and latency, and discusses using preference predictions to interpret model strengths and weaknesses. It also outlines evaluating predictions against Chatbot Arena and building a preference predictor. ### Source excerpt A challenge of building AI applications is choosing which model to use. What if we don't have to? What if we can predict the best model for any prompt? Predictive human preference aims to predict which model users might prefer for a specific query. Human preference has emerged to be both the Northstar and a powerful tool for AI model development. Human preference guides post-training techniques including RLHF and DPO. Human preference is also used to rank AI models, as used by LMSYS's Chatbot Arena. Chatbot Arena aims to determine which model is generally preferred. I wanted to see if it's possible to predict which model is preferred for each query. One use case of predictive human preference is model routing. For example, if we know in advance that for a prompt, users will prefer Claude Instant's response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency. Another use case of predictive human preference is interpretability. Mapping out a model's performance on different prompts can help us understand this model's strengths and weaknesses. See section Experiment results for examples. Here's what predictive human preference for different model pairs looks like for the prompt "What's the best way to cluster text embeddings?". The predictions were generated by my toy preference predictor. The bright yellow color for the (GPT-4, GPT-3.5-Turbo) cell means that my predictor thinks GPT-4's response is very likely to be preferred to that of GPT-3.5-Turbo's for this prompt. This post first discusses the correctness of Chatbot Arena, which will then be used as a baseline to evaluate the correctness of preference predictions. It then discusses how to build a preference predictor and the initial results. Ranking Models Using Human Preference Using preferential signals (comparisons) to rank models has grown in popularity in the last