# Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

DevFeed: [Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers](<https://devfeed.tech/articles/not-all-loops-cost-the-same-phase-dependent-depth-for-serving-looped-transformers-121266.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07211>)

Author: Zhixu Du, Yuhan Cheng, Hai "Helen" Li, Yiran Chen

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [all](<https://devfeed.tech/tags/all.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [better](<https://devfeed.tech/tags/better.md>), [budget](<https://devfeed.tech/tags/budget.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [changes](<https://devfeed.tech/tags/changes.md>), [controller](<https://devfeed.tech/tags/controller.md>), [core](<https://devfeed.tech/tags/core.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [demand](<https://devfeed.tech/tags/demand.md>), [loops](<https://devfeed.tech/tags/loops.md>), [serving](<https://devfeed.tech/tags/serving.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

LoopDuet reduces the cost coupling between prefilling and decoding in looped transformers by jointly selecting separate recurrence budgets while preserving decoding depth at prefilling boundaries. Its vLLM implementation batches tokens needing further recurrence using existing weights and attention kernels; a queue-age controller adapts prefilling depth as demand changes. On HotpotQA with one A100-80GB, Huginn achieved 19% more throughput than the fastest tested uniform setting meeting the same accuracy criterion. Another operating point gained 6.04 EM points over uniform depth eight at 97% of its throughput. On two OLMo arrival traces, adaptive prefilling gained 0.92-1.35 EM points over the most accurate tested fixed-depth controls while meeting a one-second time-to-first-token target in all cases. These results are limited to the reported setups.

## Source excerpt

arXiv:2611.07211v1 Announce Type: new Abstract: Looped transformers scale inference-time reasoning by repeatedly reusing a shared core, increasing computation without increasing parameter count. Serving prefilling and decoding with a uniform loop budget couples their costs. We present LoopDuet, a phase-dependent depth and training-free serving approach that selects the two budgets jointly while preserving decoding depth at prefilling boundaries. Our vLLM implementation gathers tokens requiring further recurrence into compact batches, turning the selected allocation into executable savings with existing weights and attention kernels. At runtime, a queue-age controller adjusts prefilling depth to retain quality as demand changes. Experiments on the full HotpotQA dataset with one A100-80GB demonstrate two benefits of this flexibility. First, separate phase budgets improve the quality--throughput trade-off: Huginn delivers 19\% more throughput than the fastest tested uniform setting meeting the same accuracy criterion, while a higher-throughput operating point gains 6.04 EM points over uniform depth eight at 97\% of its throughput. Second, the same execution mechanism retains more quality under load: on two evaluated OLMo arrival traces, adaptive prefilling gains 0.92--1.35 EM points over the most accurate tested fixed-depth controls at the same 100\% attainment of a one-second time-to-first-token target. Together, these results establish LoopDuet as a practical way to turn flexible model computation into higher serving capacity and better quality under load.