# Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference](<https://devfeed.tech/articles/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference-4740.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/>)

Author: Kareem Syed-Mohammed

Published: 2026-09-10T21:58:09Z

Content type: release

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Low-Latency Inference](<https://devfeed.tech/topics/low-latency-inference.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [caching](<https://devfeed.tech/tags/caching.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [routing](<https://devfeed.tech/tags/routing.md>), [tensorrt-llm](<https://devfeed.tech/tags/tensorrt-llm.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

## AI overview

Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token.

## Source excerpt

Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.