# Express Language Modeling

DevFeed: [Express Language Modeling](<https://devfeed.tech/articles/express-language-modeling-108512.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.10944>)

Author: Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi, Lester Mackey

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [context window](<https://devfeed.tech/topics/context-window.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [error](<https://devfeed.tech/tags/error.md>), [express](<https://devfeed.tech/tags/express.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [i-o](<https://devfeed.tech/tags/i-o.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language](<https://devfeed.tech/tags/language.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [memory](<https://devfeed.tech/tags/memory.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>)

## AI overview

Express converts non-causal attention approximations into causal ones while preserving their approximation guarantees. Paired with Thinformer, it reports approximation error of log³⁄²(n)/s, O(s) memory, and O(s² log²(n)) compression overhead for sequence length n. The authors also describe an I/O-aware Triton implementation with substantial speedups over FlashAttention 2 and apply Express to long-context prefill, KV-cache compression, and long-form decoding under memory or compute constraints. The excerpt does not quantify the speedups or detail the four applications.

## Source excerpt

arXiv:2606.10944v3 Announce Type: replace Abstract: We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering $\log^{3/2}(n)/s$ approximation error with only $O(s)$ memory and $O(s^2 \log^2(n))$ compression overhead for a sequence of length $n$. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.