# Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs

DevFeed: [Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs](<https://devfeed.tech/articles/twig-parallel-and-memory-efficient-tree-speculative-decoding-for-hybrid-attention-llms-120464.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.01651>)

Author: Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [batch](<https://devfeed.tech/tags/batch.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [integrate](<https://devfeed.tech/tags/integrate.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [llms](<https://devfeed.tech/tags/llms.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [models](<https://devfeed.tech/tags/models.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [twig](<https://devfeed.tech/tags/twig.md>)

## AI overview

Twig addresses hybrid LLMs' parent-dependent recurrent linear-attention states, which force existing engines to verify speculative trees sequentially and retain full state snapshots per node. It factors the recurrence into a tree structure and uses a GPU kernel to verify nodes in parallel, reporting 3.4-7.7x faster linear-attention tree verification. Token-level factors let it reconstruct only the accepted state, reducing transient state memory by 82-99x. Integrated into SGLang, Twig also uses batch-wide verification budgeting. Across four models and two GPU platforms, it reports up to 4.72x offline decode throughput versus autoregressive decoding and up to 2.03x versus the strongest tree-speculative baseline. In online agent workloads, TTFT and TPOT fall by up to 67.6% and 49.9%, respectively, versus that baseline.

## Source excerpt

arXiv:2608.01651v2 Announce Type: replace-cross Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.