# Small Language Models for Inference-Capability Routing: Effects of Model Capacity and Prompt Optimization on Downstream Quality and Efficiency

DevFeed: [Small Language Models for Inference-Capability Routing: Effects of Model Capacity and Prompt Optimization on Downstream Quality and Efficiency](<https://devfeed.tech/articles/small-language-models-for-inference-capability-routing-effects-of-model-capacity-and-prompt-optimization-on-downstream-quality-and-efficiency-132231.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16414>)

Author: Sailendra Kolluru, Maheshkumar Duvvarapu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [quality](<https://devfeed.tech/tags/quality.md>), [routing](<https://devfeed.tech/tags/routing.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [token](<https://devfeed.tech/tags/token.md>)

## AI overview

The study tests prompt-driven Qwen2.5 routers (1.5B-14B parameters) that independently select reasoning, retrieval, web search, and tool use, allowing multiple capabilities per query. Across five prompting strategies, added capacity does not consistently improve classification under minimal prompts. Prompt optimization narrows the capacity gap for reasoning and calculator routing, while RAG and web search retain larger benefits from higher capacity. On 127 questions using GPT-OSS-120B, the 14B router's mean BERTScore F1 (0.8408) is close to enabling all capabilities (0.8418); it cuts average downstream tokens by 68.7% and median downstream latency by 80.5%. These savings exclude router overhead, which was measured separately in a different inference environment and increases with router capacity. The abstract cautions that similar classification F1 can mask different error costs: false positives waste computation, while missed capabilities can lower answer quality.

## Source excerpt

arXiv:2611.16414v1 Announce Type: new Abstract: Optional reasoning, retrieval, web search, and tool use can improve large language model responses but incur unnecessary computation when activated indiscriminately. This study investigates whether prompt optimization can compensate for limited router capacity and whether classification gains translate into downstream efficiency. The proposed framework uses prompt-driven small language models to make four independent capability decisions, allowing multiple capabilities to be selected per query. Qwen2.5 routers ranging from 1.5B to 14B parameters are compared across five prompting strategies, followed by an evaluation of capability-specific optimized configurations using a fixed GPT-OSS-120B downstream model on 127 questions. Under minimal prompting, increasing router capacity does not consistently improve classification performance. Prompt optimization narrows the capacity gap for reasoning and calculator routing, whereas RAG and web search retain larger benefits from higher capacity. Similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality. The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7\% (8,837 to 2,764 tokens) and median downstream latency by 80.5\% (9.79 to 1.91 seconds). These savings exclude router overhead, which is measured separately in a different inference environment and increases with router capacity. The findings demonstrate that the relationship between router capacity, prompt design, and downstream efficiency is capability-dependent. Consequently, router selection must consider error profiles and routing overhead alongside classification performance.