# Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling

DevFeed: [Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling](<https://devfeed.tech/articles/fairway-load-aware-kv-cache-reuse-for-llm-prefill-scheduling-113808.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02082>)

Author: Hamidreza Alikhani, Nazmul Takbir, Matin Khajavi, Rajat Bhattacharjya, Sangeetha Abdu Jyothi, Nikil Dutt

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load](<https://devfeed.tech/tags/load.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [rag](<https://devfeed.tech/tags/rag.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [serving](<https://devfeed.tech/tags/serving.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>), [ttft](<https://devfeed.tech/tags/ttft.md>)

## AI overview

Fairway addresses the tradeoff between KV reuse-first scheduling, which lowers median time-to-first-token (TTFT) but can starve requests with more uncached work during load spikes, and FCFS, which bounds the tail while raising the median. It uses a virtual dual-lane model: a reuse-first Fast lane speeds up cache-hit-heavy requests, while a Fair lane shifts allocation toward requests that have waited longer relative to their own prefill time. The scheduler adapts without per-workload tuning. According to the abstract, across tested workloads and load levels it matches or beats the best baseline, with p99 TTFT up to 8.6x lower than prior prefix-aware schedulers; during spikes it retains FCFS's tail behavior with a 6x lower median TTFT.

## Source excerpt

arXiv:2611.02082v1 Announce Type: new Abstract: LLM-based applications often employ agentic workflows and retrieval-augmented generation (RAG), which build long prompts from shared context. When prefixes are shared across requests, serving engines reuse the key-value (KV) cache to skip parts of the prefill computation. Reuse is typically graded: the uncached work of requests in a queue can differ by up to 100x. In this setting, the order of scheduling determines the time-to-first-token (TTFT) for the queued requests. KV reuse-first scheduling, which prioritizes the requests with the longest KV cache hit, gives the lowest median TTFT but starves expensive requests under load spikes; first-come-first-served (FCFS) scheduling bounds the tail but raises the median TTFT. We present Fairway, a prefill scheduler built on adaptive reuse prioritization: it favors KV cache reuse while queues are short, and shifts toward fair allocation as requests wait longer relative to their own prefill time. Fairway employs a virtual dual lane model: a KV reuse-first Fast lane and a Fair lane atop the serving engine's waiting queue that enables speeding up of cheap reuse requests without starving expensive ones. This model automatically adapts to the workload without any per-workload tuning. Fairway lowers the tail TTFT at no cost to the median in transient overload. Across workloads and load levels, Fairway matches or beats the best baseline, with a p99 TTFT up to 8.6x lower than prior prefix-aware schedulers. During load spikes, it keeps the tail of FCFS at a 6x lower median TTFT.