# infini-attention

Infini-attention is an attention technique for Transformer-based large language models that incorporates compressive memory with masked local and long-term linear attention mechanisms in a single Transformer block.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention

DevFeed: [Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention](<https://devfeed.tech/articles/attention-sinks-as-spectral-spikes-a-mechanism-analysis-of-gated-attention-137569.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20403>)

Author: Seojin Kim, Yehjin Shin, Noseong Park

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [attention-sink](<https://devfeed.tech/tags/attention-sink.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [outlier](<https://devfeed.tech/tags/outlier.md>), [rank](<https://devfeed.tech/tags/rank.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The paper models attention sinks as low-rank modes in token-space covariance aligned with high-mass attention columns. It argues that post-SDPA output gating contracts these modes: in the forward pass, it attenuates sink-related spectral energy; in the backward pass, it locally reduces value-path gradients routed through sink columns. The reported diagnostics show reduced spectral outlier mass and sink-subspace alignment, increased effective rank, and attenuation of first-token value gradients when the gate is active. During training, gating also suppresses the co-emergence of concentrated attention, amplified value gradients, and dominant representations. The evidence provided is an abstract, so details of the experiments are not available here.

### Source excerpt

arXiv:2611.20403v1 Announce Type: cross Abstract: Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.

## Temporal Context-Aware Spam Detection Using Transformer-Based NLP Models

DevFeed: [Temporal Context-Aware Spam Detection Using Transformer-Based NLP Models](<https://devfeed.tech/articles/temporal-context-aware-spam-detection-using-transformer-based-nlp-models-133084.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16139>)

Author: Fariz Zeynalov, Razim Mammadli

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [text-classification](<https://devfeed.tech/topics/text-classification.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [spam](<https://devfeed.tech/tags/spam.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The abstract reports outbound SMS spam detection at send time using 43,741 messages from a proprietary production database. On an 8,749-message held-out test set, text-only DeBERTa-v3-base outperformed TF-IDF with logistic regression in AUC-ROC (0.9011 vs. 0.8721); DeBERTa's F1 was 0.8412. A deployed DistilBERT model with inter-message temporal features achieved 0.8238 accuracy, 0.8382 spam-class F1, and 0.9128 recall at threshold 0.5 on a separate, class-balanced 20,000-message production evaluation. Those figures are not directly comparable to the held-out results. The paper also describes TemporalBertSpam, which uses a causal Transformer to encode each sender's message history, but says a same-split evaluation is needed to measure its contribution.

### Source excerpt

arXiv:2611.16139v1 Announce Type: cross Abstract: Spam detection is usually treated as single-message classification, ignoring behavioral patterns that emerge across sequences of messages from the same sender. We study outbound SMS spam detection at the carrier level, where the goal is to predict at send time whether a carrier will flag a message as spam. Using a proprietary production database of 43,741 outbound SMS messages with UTC transmission timestamps and carrier delivery statuses, we report three sets of results. On an 8,749-message held-out test set, a TF-IDF + logistic regression baseline reaches AUC-ROC 0.8721, and a text-only DeBERTa-v3-base classifier reaches AUC-ROC 0.9011 and F1 0.8412. A deployed DistilBERT model augmented with inter-message temporal features reaches accuracy 0.8238, spam-class F1 0.8382, and recall 0.9128 at threshold 0.5 on a separate, class-balanced 20,000-message production evaluation set; these results are not directly comparable with the held-out test results. We also describe TemporalBertSpam, an architecture that encodes each sender's message history with a causal Transformer, and outline the same-split evaluation needed to measure its contribution.

## Everything in its right place: attention is a radio head

DevFeed: [Everything in its right place: attention is a radio head](<https://devfeed.tech/articles/everything-in-its-right-place-attention-is-a-radio-head-128261.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13146>)

Author: Llu\\'is Arola-Fern\\'andez, Aissam Khadraoui, Alex Arenas

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [antenna](<https://devfeed.tech/tags/antenna.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [context](<https://devfeed.tech/tags/context.md>), [count](<https://devfeed.tech/tags/count.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [device](<https://devfeed.tech/tags/device.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [nlin-ao](<https://devfeed.tech/tags/nlin-ao.md>), [phases](<https://devfeed.tech/tags/phases.md>), [physics-soc-ph](<https://devfeed.tech/tags/physics-soc-ph.md>), [position](<https://devfeed.tech/tags/position.md>), [query](<https://devfeed.tech/tags/query.md>), [radio](<https://devfeed.tech/tags/radio.md>), [receiver](<https://devfeed.tech/tags/receiver.md>), [select](<https://devfeed.tech/tags/select.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>), [theory](<https://devfeed.tech/tags/theory.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The paper models attention as synchronization between a receiver and context-token oscillators: token content aligns the receiver's antenna with the query, while token age sets its frequency. Its closed-form layer has two regimes: above a context-dependent critical coupling, the receiver locks and weights normalize; below it, the receiver stays silent and weights sum to less than one. The authors report that trained networks use these regimes to select, abstain and count, fail where the theory predicts, and read well beyond their training length.

### Source excerpt

arXiv:2611.13146v1 Announce Type: cross Abstract: Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.

## Transferring local rib responses to new component geometries in sheet forming with a graph-based Transformer

DevFeed: [Transferring local rib responses to new component geometries in sheet forming with a graph-based Transformer](<https://devfeed.tech/articles/transferring-local-rib-responses-to-new-component-geometries-in-sheet-forming-with-a-graph-based-transformer-125132.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10086>)

Author: Yingxue Zhao, Haoran Li, Haosu Zhou, Tobias Pfaff, Nan Li

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [component](<https://devfeed.tech/tags/component.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [graph](<https://devfeed.tech/tags/graph.md>), [local](<https://devfeed.tech/tags/local.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [transfer-learning](<https://devfeed.tech/tags/transfer-learning.md>)

### AI overview

R-BiLoGo learns the rib-induced changes in displacement and thinning relative to a baseline, then transfers that local-feature response across hot-stamped corner geometries. On the source geometry and two of three selected targets, it had lower errors than hierarchical graph-based and Transolver-based models. Across 22 targets, transfer using 20 target simulations matched or beat training from scratch with 80 simulations for about half the targets; with 80 simulations, it reduced errors for 95%. Transfer benefits weakened as source and target differed more in surface distance or high-curvature geometry, and when targets were shallower than the source. These trends also appeared in other corner types, with different magnitudes.

### Source excerpt

arXiv:2611.10086v1 Announce Type: new Abstract: Sheet metal components often consist of a global geometry, such as the corner radii and forming depth of a corner component, and local features added to it. During forming, both affect material flow. A surrogate learning all their combinations may need large training data. However, the same local feature type is often used on many global geometries. This study therefore partly decouples local features from the global geometry. The local-feature effect is first learned on one global geometry, needing fewer simulations than learning all combinations, then transferred to other global geometries via transfer learning, which can need fewer target simulations than training from scratch. A baseline-conditioned residual surrogate, R-BiLoGo, is proposed for ribs on the side walls of hot-stamped corner components. Its inputs are the baseline response and the geometric perturbation introduced by the ribs. Its outputs are the changes in displacement and thinning from the baseline. R-BiLoGo gave lower errors than a hierarchical graph-based model and a Transolver-based model on the source geometry, and on two of three selected target geometries after transfer. Across the 22 target geometries, transfer learning with 20 target simulations gave errors similar to or lower than training from scratch with 80 target simulations for about half of the targets. With the same 80 target simulations, transfer learning reduced errors for 95% of targets. The effect of the global geometry on transfer was studied with four parameter-free geometric descriptors and controlled variations across 4 corner subclasses. The benefit of transfer decreased with increasing surface distance and high-curvature change between source and target, and for targets shallower than the source. These trends were also seen in other corner types with a different magnitude.

## Know What You Need: Efficient Activation Checkpointing

DevFeed: [Know What You Need: Efficient Activation Checkpointing](<https://devfeed.tech/articles/know-what-you-need-efficient-activation-checkpointing-123431.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09614>)

Author: Danyal Ahmed (MLIA), Edouard Oyallon (MLIA)

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [memory](<https://devfeed.tech/tags/memory.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The method reduces activation-rematerialization compute by using information from the initial forward pass to identify which layer components to recompute, via greedy selection or randomized sketching. It aims to produce unbiased gradient estimates; ablations report gradients close to exact ones. In Llama-488M experiments, recomputing two-thirds of components in feed-forward layers, attention layers, or both raised perplexity by 1.7%, 1.2%, and 2.5%, respectively. The supplied abstract does not report broader evaluation or implementation details.

### Source excerpt

arXiv:2611.09614v1 Announce Type: new Abstract: Large Language Model training is increasingly constrained by memory capacity. Activation Checkpointing reduces activation memory by avoiding storing selected activations during the forward pass and rematerializing them during backpropagation, thereby trading memory savings for additional computation. Although this technique is essential for scaling model training, rematerialization can be prohibitively expensive, particularly for Transformer linear layers. Approximate computation could reduce this overhead, but conventional approaches either introduce errors that accumulate over the backward pass or fail to exploit information available from the initial forward pass. We introduce a novel approximate rematerialization method that can yield unbiased gradient estimates: using information collected during the initial forward pass, the method selectively recomputes only the most important layer components, either through greedy selection or randomized sketching. Ablation studies show that the resulting gradients closely match the exact gradients. When applied to feed-forward layers, attention layers, or both, our method can recompute only two-thirds of the layer components in a Llama-488M model, while increasing perplexity by just 1.7%, 1.2%, and 2.5%, respectively.

## Local-Frame Rotary Attention for Antibody-Specific Epitope Prediction

DevFeed: [Local-Frame Rotary Attention for Antibody-Specific Epitope Prediction](<https://devfeed.tech/articles/local-frame-rotary-attention-for-antibody-specific-epitope-prediction-120067.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.01092>)

Author: Chuanliu Fan, Junjie Wu, Nan Yu, Guohong Fu

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [frame](<https://devfeed.tech/tags/frame.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [local](<https://devfeed.tech/tags/local.md>), [prediction](<https://devfeed.tech/tags/prediction.md>)

### AI overview

Local-Frame Rotary Attention (LFRA) uses folded protein geometry as the positional mechanism in attention for antibody-specific epitope prediction. It expresses inter-residue displacements in backbone-defined local frames and injects them directly into rotary attention, preserving continuous directional geometry while remaining invariant to independent SE(3) transformations. The abstract contrasts this with methods that add graph, surface, or point-cloud encoders while retaining attention positional mechanisms tied largely to sequence order. The supplied excerpt gives no performance results or comparison benchmarks.

### Source excerpt

arXiv:2608.01092v2 Announce Type: replace Abstract: Antibody-specific epitope prediction aims to identify which antigen residues are recognized by a given antibody, a task that depends on the three-dimensional complementarity between antibody CDRs and the antigen surface. Existing methods usually leverage PLM embeddings and inject structure through additional graph, surface, or point-cloud encoders, where the positional mechanism inside attention remains largely tied to one-dimensional sequence order. For proteins, the analogue of a token offset is not only sequence separation, but also the three-dimensional displacement between residues after folding. This raises a question, can folded residue geometry serve as the positional mechanism of attention itself? We propose Local-Frame Rotary Attention (LFRA), which expresses inter-residue displacements in backbone-defined local frames and injects them directly into rotary attention. This design preserves continuous directional geometry while ensuring invariance to independent $\mathrm{SE}(3)$ transformations. The code and dataset are available at https://github.com/better-fcl/lfra.

## HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

DevFeed: [HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention](<https://devfeed.tech/articles/hisa-efficient-hierarchical-indexing-for-fine-grained-sparse-attention-116680.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.28458>)

Author: Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, Tongxuan Liu, Di Yin, Xing Sun, Muhan Zhang

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [block](<https://devfeed.tech/tags/block.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [dsa](<https://devfeed.tech/tags/dsa.md>), [fine](<https://devfeed.tech/tags/fine.md>), [indexer](<https://devfeed.tech/tags/indexer.md>), [indexing](<https://devfeed.tech/tags/indexing.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [key](<https://devfeed.tech/tags/key.md>), [layer](<https://devfeed.tech/tags/layer.md>), [mla](<https://devfeed.tech/tags/mla.md>), [operator](<https://devfeed.tech/tags/operator.md>), [pattern](<https://devfeed.tech/tags/pattern.md>), [plug](<https://devfeed.tech/tags/plug.md>), [quality](<https://devfeed.tech/tags/quality.md>), [query](<https://devfeed.tech/tags/query.md>), [regions](<https://devfeed.tech/tags/regions.md>), [replacement](<https://devfeed.tech/tags/replacement.md>)

### AI overview

HISA reduces the indexer bottleneck in fine-grained sparse attention by replacing a full-prefix token scan with two stages: it scores pooled block representations to filter out regions, then applies the original token indexer only within retained blocks. It preserves the token-level top-sparse pattern used by the downstream Sparse MLA operator and requires no additional training. The abstract reports kernel-level speedup at 64K context, but omits the amount. On Needle-in-a-Haystack and LongBench, replacing the indexer in DeepSeek-V3.2 and GLM-5 without finetuning reportedly comes close to the original DSA quality and outperforms block-sparse baselines.

### Source excerpt

arXiv:2603.28458v5 Announce Type: replace Abstract: Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a lightweight indexer, then computing attention only on the selected subset. While the downstream sparse attention itself scales favorably, the indexer must still scan the entire prefix for every query, introducing an per-layer bottleneck that grows prohibitively with context length. We propose HISA (Hierarchical Indexed Sparse Attention), a plug-and-play replacement for the indexer that rewrites the search path from a flat token scan into a two-stage hierarchical procedure: (1) a block-level coarse filtering stage that scores pooled block representations to discard irrelevant regions, followed by (2) a token-level refinement stage that applies the original indexer exclusively within the retained candidate blocks. HISA preserves the identical token-level top-sparse pattern consumed by the downstream Sparse MLA operator and requires no additional training. On kernel-level benchmarks, HISA achieves up to speedup at 64K context. On Needle-in-a-Haystack and LongBench, we directly replace the indexer in DeepSeek-V3.2 and GLM-5 with our HISA indexer, without any finetuning. HISA closely matches the original DSA in quality, while substantially outperforming block-sparse baselines.

## Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

DevFeed: [Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms](<https://devfeed.tech/articles/exact-fusion-of-softmax-attention-heads-can-require-exponentially-many-atoms-114754.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02943>)

Author: Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>)

Tags: [atom](<https://devfeed.tech/tags/atom.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper proves that exactly fusing independently normalized softmax attention heads can require exponentially more key-value atoms: for its example, d heads with two atoms each use 2d head-atom terms, while any equivalent single head requires exactly 2^d atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for uniform error below 1/2 over all queries, and exact equality on any nonempty open query domain also inherits it. The abstract also reports a tight fixed-head-count bound when the head count divides d, a normalized family with exponential exact minimum despite approaching a constant on fixed-radius query balls, and pretrained-cache results: an affine-rank bound and certified instance rule out exact preservation at a specified atom count. Learned-cache comparisons across two disjoint 64-context banks show budget-dependent reversals and calibration effects.

### Source excerpt

arXiv:2611.02943v1 Announce Type: new Abstract: Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W_d(q)=\sum_{j=1}^d\sigma(q_j)$: its $d$ two-atom heads use $2d$ head-atom terms, whereas its minimum single-head representation has exactly $2^d$ atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below $1/2$ on $\mathbb R^d$, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides $d$. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.

## Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer

DevFeed: [Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer](<https://devfeed.tech/articles/nonconformal-distortion-and-local-orientation-reversal-in-the-fourier-circuit-of-a-modular-addition-transformer-114378.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00414>)

Author: Eduardo Fuentes, Gonzalo Riera, H\\'ector Allende-Cid

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [fourier analysis](<https://devfeed.tech/topics/fourier-analysis.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [mlp](<https://devfeed.tech/topics/mlp.md>), [fourier transform](<https://devfeed.tech/topics/fourier-transform.md>)

Tags: [addition](<https://devfeed.tech/tags/addition.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [local](<https://devfeed.tech/tags/local.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [model](<https://devfeed.tech/tags/model.md>), [modular](<https://devfeed.tech/tags/modular.md>), [neural](<https://devfeed.tech/tags/neural.md>), [orientation](<https://devfeed.tech/tags/orientation.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The study measures local deformation in the MLP component of a Fourier-based modular-addition circuit in a one-layer transformer. Across five key-frequency input planes, trained maps show nonconformal distortion (median |μ|=0.304-0.446), spatial structure in the Beltrami argument, and local orientation reversals, despite ideal complex multiplication being conformal in nondegenerate restrictions. Median distortion falls from 0.939 at initialization to 0.390 after training. Key-frequency planes have lower median distortion than most random-plane controls, though distributions overlap and coordinate-frame conditioning differs. In one training run, most aggregate distortion reduction precedes the main rise in measured circuit formation; distortion has negligible association with correct-logit margin (|ρ|≤0.026). Late-checkpoint distortion is comparable across six runs for most selected frequencies. The analysis characterizes a frequency-projected circuit component and complements its algorithmic reconstruction.

### Source excerpt

arXiv:2611.00414v1 Announce Type: cross Abstract: We study the local differential geometry of a Fourier-based modular-addition circuit in a one-layer transformer whose algorithm has previously been reverse-engineered. For each of the five key Fourier frequencies, we restrict the model's multilayer perceptron (MLP) contribution to a corresponding Fourier input plane and readout, and measure its directional Beltrami coefficient under continuous input perturbations. Whereas nondegenerate restrictions of ideal complex multiplication are conformal, the trained maps exhibit nonconformal distortion (median $|\mu|=0.304$--$0.446$), spatial organization in the Beltrami argument, and regions of local orientation reversal. The median across frequencies decreases from $0.939$ at initialization to $0.390$ after training. The key-frequency planes also show lower median distortion than most random-plane controls, although their distributions overlap and the coordinate frames differ in conditioning. Across 500 checkpoints in one training run, most of the aggregate reduction in Beltrami magnitude occurs before the principal increase in the measured Fourier-circuit formation. At the individual-input level, Beltrami magnitude shows negligible association with the correct-logit margin ($|\rho|\leq0.026$). Late-checkpoint measurements across six runs show comparable distortion for most selected frequencies. These results provide a quantitative characterization of local deformation in a frequency-projected component of an identified neural circuit, complementary to its algorithmic reconstruction.

## A Positional Realizability Theory of Length Generalization in Single-Head Attention

DevFeed: [A Positional Realizability Theory of Length Generalization in Single-Head Attention](<https://devfeed.tech/articles/a-positional-realizability-theory-of-length-generalization-in-single-head-attention-111558.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41218>)

Author: Jiasong Han, Yunkai Chai, Jintao Wang

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [absolute](<https://devfeed.tech/tags/absolute.md>), [addition](<https://devfeed.tech/tags/addition.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [lg](<https://devfeed.tech/tags/lg.md>), [positional-encoding](<https://devfeed.tech/tags/positional-encoding.md>), [theory](<https://devfeed.tech/tags/theory.md>)

### AI overview

The paper explains length generalization in a controlled model of a single positional-only attention head. Its transfer operator evaluates a positional kernel learned at one sequence length on a longer grid; a pretraining scalar α_L provides a sufficient-condition witness for generalization and separates generalizing from non-generalizing positional-encoding/task cells in 2,625 trained-model measurements. The results include a characterization for relative heads under a strict unique-global-maximum condition, an absolute/relative duality, and limits on learned-absolute determination and static expressivity. The general deterministic case remains an open conjecture. The theory concerns positional information alone: content-dependent tasks such as addition and sorting fall outside the stated characterization.

### Source excerpt

arXiv:2610.41218v1 Announce Type: new Abstract: Why does length generalization (LG) depend so sharply on the positional encoding (PE)? Relative schemes extrapolate fixed offsets, absolute schemes fix positions, and neither handles growing offsets. We give a realizability theory for a deliberately controlled abstraction, a single positional-only attention head, that derives this well-documented empirical pattern and says precisely where it holds and breaks. The device is a simulation transfer operator $T_L$ that re-evaluates a selected length-$L_0$-realizing positional kernel on the longer grid, with a scalar $\alpha_L \in [0,1]$ read off before any training from $(\Phi, t_n)$ alone. Here $\alpha_L$ serves as the computable witness of that taxonomy's derivation, not as a stronger empirical predictor. Our results are a relative-head deep-row characterization for kernels with a strict unique global maximizer (Theorem 2), an absolute/relative duality that derives the pattern (Proposition 4), a learned-absolute underdetermination result, and a static expressivity bound; a sufficiency theorem (Theorem 1) bridges $\alpha_L$ to LG, and the general deterministic case is a precise open conjecture (correcting a tempting but invalid "outside-colspan" argument). Experiments confirm the structure: in the controlled single-head regime $\alpha_L$ exactly separates the taxonomy's generalizing and non-generalizing (PE, task) cells (2625 trained-model measurements; stable at vocab 64/d=128). Together, these results turn the received PE taxonomy into a positional-kernel transfer theorem, and identify the content and depth mechanisms that escape it. The theory concerns the positional channel alone: tasks whose targets depend on token content, including the standard length-generalization suite such as addition and sorting, fall outside Theorem 2.

## Subquadratic Attention with a Hierarchical Tree-Structured Indexer

DevFeed: [Subquadratic Attention with a Hierarchical Tree-Structured Indexer](<https://devfeed.tech/articles/subquadratic-attention-with-a-hierarchical-tree-structured-indexer-112145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.42112>)

Author: Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy, Tom\\'as Figliolia, Beren Millidge

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [peft](<https://devfeed.tech/topics/peft.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [based](<https://devfeed.tech/tags/based.md>), [but](<https://devfeed.tech/tags/but.md>), [cache](<https://devfeed.tech/tags/cache.md>), [call](<https://devfeed.tech/tags/call.md>), [can](<https://devfeed.tech/tags/can.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context](<https://devfeed.tech/tags/context.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [deep](<https://devfeed.tech/tags/deep.md>), [indexer](<https://devfeed.tech/tags/indexer.md>), [structured](<https://devfeed.tech/tags/structured.md>), [subquadratic](<https://devfeed.tech/tags/subquadratic.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Zyphra Hierarchical Attention (ZHA) aims to reduce attention's quadratic cost while retaining sparse access to the KV cache. It builds recursively compressed key summaries in a tree, which queries traverse top-down to select keys; increasing tree depth makes routing plus final attention subquadratic, though still superlinear, in sequence length. The authors report that ZHA can be applied to pretrained models with a short KL-distillation stage over a few billion tokens, maintains performance on retrieval, reasoning, math, and agentic coding tasks, and tolerates tree geometries up to six levels. Their performance model reports advantages over competing methods beyond one million tokens; context windows of hundreds of millions or billions are presented as potential, not demonstrated outcomes.

### Source excerpt

arXiv:2610.42112v1 Announce Type: cross Abstract: The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention's expressiveness (e.g., sliding-window or linear attention variants with fixed-size state) or only achieve constant-factor speedups (e.g., sparse attention methods whose indexers themselves scale quadratically). In this paper, we derive an asymptotically subquadratic sparse attention method based on a hierarchical, tree-structured indexer, which we call Zyphra Hierarchical Attention (ZHA). ZHA constructs a tree of recursively compressed key summaries over the KV cache, which future queries traverse from the top down to retrieve a sparse subset of keys. Crucially, by iteratively growing the tree's depth, the total cost including routing and final attention grows sublinearly in sequence length, resulting in overall subquadratic (but superlinear) scaling for attention. We demonstrate that ZHA can be cheaply applied to existing pretrained models with only a short KL distillation training stage over a few billion tokens, and maintains performance on long-context retrieval and challenging reasoning, mathematical, and agentic coding evaluation tasks. Moreover, we show that our method is highly robust to tree geometry including very deep trees of up to 6 levels. Finally, we develop a detailed performance model and show that our approach scales subquadratically, dramatically outperforms competing methods beyond 1 million context, and ultimately has the potential to extend effective context windows to the hundreds of millions or billions of tokens.

## ASTROLABE: A 3D Particle-Level Surrogate for Cosmology Simulation with Physical Flow Matching

DevFeed: [ASTROLABE: A 3D Particle-Level Surrogate for Cosmology Simulation with Physical Flow Matching](<https://devfeed.tech/articles/astrolabe-a-3d-particle-level-surrogate-for-cosmology-simulation-with-physical-flow-matching-103027.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.27068>)

Author: Ibrahim Elsharkawy, David Curtin, Yonatan Kahn, Neal Weiner

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Simulation](<https://devfeed.tech/topics/simulation.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [astro-ph-co](<https://devfeed.tech/tags/astro-ph-co.md>), [astrolabe](<https://devfeed.tech/tags/astrolabe.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dark-matter](<https://devfeed.tech/tags/dark-matter.md>), [emulation](<https://devfeed.tech/tags/emulation.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [generative](<https://devfeed.tech/tags/generative.md>), [hep-ph](<https://devfeed.tech/tags/hep-ph.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [momentum](<https://devfeed.tech/tags/momentum.md>), [out-of-distribution-generalization](<https://devfeed.tech/tags/out-of-distribution-generalization.md>), [power](<https://devfeed.tech/tags/power.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [space](<https://devfeed.tech/tags/space.md>), [structure](<https://devfeed.tech/tags/structure.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [velocity](<https://devfeed.tech/tags/velocity.md>), [voxel](<https://devfeed.tech/tags/voxel.md>)

### AI overview

Astrolabe emulates cosmological simulations by generating the 6N phase-space coordinates of individual particles, avoiding the voxel-resolution ceiling of grid-based outputs and allowing observables to be measured from inferred particles. Its field-particle hybrid transformer uses multiscale field tokens with compute and memory linear in particle count. The flow starts from approximate COLA-based coordinates, with momentum and mass conservation enforced throughout. In tested Lambda-CDM cases, it reproduces power spectra, halo mass functions, velocity profiles, and halo density profiles, and the abstract reports accurate emulation of dark-acoustic oscillation initial conditions. Fine-tuning also enables a TNG50-3-dark simulation at a fraction of the cost. The authors describe this as a first step; adding baryonic physics and exploring richer dark-sector interactions remain potential future directions.

### Source excerpt

arXiv:2610.27068v1 Announce Type: cross Abstract: The microphysics of dark matter (DM) is imprinted on cosmological structure across an enormous dynamic range. Nonlinear cosmological observables are most reliably modeled with N-body and hydrodynamic simulations. However, systematic N-body scans over DM theory space are prohibitively expensive. Machine learning emulators provide a possible solution, but are currently limited by either low resolution or only the ability to emulate specific observables. We introduce Astrolabe, a flow-matching-based generative emulator designed to be free of both constraints. Astrolabe acts directly on the 6N phase-space coordinates of N individual particles. With no voxel grid in the output, there is no voxel-resolution ceiling, and in principle any observable can be measured from inference. Astrolabe is computationally tractable via a field-particle hybrid transformer in which every particle cross-attends to multiscale field tokens, so compute and memory are linear in N. Astrolabe incorporates physical flow matching by taking the prior distribution for the flow to be approximate COmoving Lagrangian Acceleration (COLA)-based coordinates rather than unphysical noise, with momentum and mass conservation enforced at all points along flow time. In the Lambda-CDM cases we test, Astrolabe produces realistic fields with accurate power spectra, halo mass functions, velocity profiles, and halo density profiles. We also observe out-of-distribution generalization, with accurate emulation of dark-acoustic oscillation initial conditions. Astrolabe is also capable of being fine-tuned to simulate very large N and we demonstrate this by emulating a TNG50-3-dark simulation at a fraction of the cost. This method represents a first step toward a general and powerful cosmology simulation surrogate, which may eventually permit the inclusion of baryonic physics and a systematic exploration of richer dark sector interactions.

## TreeWY: Speculative Verification for Gated DeltaNet Hybrids

DevFeed: [TreeWY: Speculative Verification for Gated DeltaNet Hybrids](<https://devfeed.tech/articles/treewy-speculative-verification-for-gated-deltanet-hybrids-99441.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.20961>)

Author: Sneha Murthy Ghantasala

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [memory](<https://devfeed.tech/tags/memory.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

TreeWY replaces per-draft-position recurrent-state snapshots in Gated DeltaNet layers with a small pseudo-value matrix. A tree-structured WY transform computes draft-node outputs using one triangular solve, then reconstructs only the accepted state when committing; the derivation depends on the gated delta rule. In serving benchmarks on Qwen3.5 35B and 397B, it reduces recurrent-state memory and KV-cache pressure with essentially unchanged acceptance length. Where memory is a bottleneck, the freed HBM enables higher throughput and much lower time-to-first-token; where it is not, the method costs a few percent. It also makes wider, higher-acceptance draft trees affordable, though the abstract reports no throughput gain from that change yet.

### Source excerpt

arXiv:2608.20961v2 Announce Type: replace-cross Abstract: Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at essentially identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.

## OpenAttention: One Percent Is Enough with Tail Proxies

DevFeed: [OpenAttention: One Percent Is Enough with Tail Proxies](<https://devfeed.tech/articles/openattention-one-percent-is-enough-with-tail-proxies-97004.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30951>)

Author: Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim, Sungjun Park, Jaemin Choi, Jincheol Yang, Seunghun Moon, Bongjoon Hyun, Kyomin Sohn, Kyeongbo Kong, Suk-Ju Kang

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [clustering](<https://devfeed.tech/topics/clustering.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [efficiently](<https://devfeed.tech/tags/efficiently.md>), [error](<https://devfeed.tech/tags/error.md>), [faster](<https://devfeed.tech/tags/faster.md>), [flash](<https://devfeed.tech/tags/flash.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [key-value-cache](<https://devfeed.tech/tags/key-value-cache.md>), [keys](<https://devfeed.tech/tags/keys.md>), [llama](<https://devfeed.tech/tags/llama.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [query](<https://devfeed.tech/tags/query.md>), [ruler](<https://devfeed.tech/tags/ruler.md>), [selection](<https://devfeed.tech/tags/selection.md>), [selector](<https://devfeed.tech/tags/selector.md>), [token](<https://devfeed.tech/tags/token.md>), [top](<https://devfeed.tech/tags/top.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

OpenAttention addresses sparse attention's unread key-value cache tail with a count-aware mean proxy, which the authors report gives lower output error than sampling at every tested read level behind a practical selector. They say sampling nearly doubles the weight error of discarding at low reads, while one proxy stays below discarding's error. Their pipeline combines training-free Query-Geometry Clustering (QGC), a Query-Geometry Selector (QGS), ErrorGate allocation based on estimated cluster weight error, and top-k selection. On RULER-32K HARD with Llama-3.1-8B, it reportedly retains 99% of dense accuracy at a 1% read; the abstract also reports 10x faster decode attention from a customized kernel at that read level. The excerpt does not provide evaluation details beyond these claims.

### Source excerpt

arXiv:2610.30951v1 Announce Type: new Abstract: Sparse attention reads only a small subset of the key-value cache, and how to treat the unread tail remains a key challenge. Discarding the tail biases the output, and sampling it as previous approaches do fails at reads of 5% and below, where the tail likely holds heavy hitters the selector missed. Our key observation is that the simplest possible proxy, one token that folds the entire tail into its count-aware mean, already yields lower output error than a sample at every read behind a practical selector. We further show in closed form that at low reads sampling nearly doubles the weight error of discarding while a single proxy stays below it. Building on these findings, we explore approximating the dense attention distribution with a few proxies that target the tail. We propose OpenAttention, a pipeline of clustering, ErrorGate and top-k selection followed by a count-aware mean proxy. To implement clustering and selection both efficiently and effectively, we design training-free Query-Geometry Clustering (QGC) and a Query-Geometry Selector (QGS), which compress the keys while respecting the query directions. Additionally, unlike prior methods that rank clusters by centroid alone, ErrorGate allocates the exact read by the estimated weight error of each cluster. On RULER-32K HARD with Llama-3.1-8B, OpenAttention retains 99% of dense accuracy at a 1% read. It also achieves state-of-the-art accuracy across models, context lengths and datasets. Our customized attention kernel, OpenAttention.flash, runs the decode attention stage 10$\times$ faster than dense attention at a 1% read.

## Reader--Writer Coupling in Transformer MLPs

DevFeed: [Reader--Writer Coupling in Transformer MLPs](<https://devfeed.tech/articles/reader-writer-coupling-in-transformer-mlps-94910.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.29340>)

Author: Giyeong Oh, Sunghee Ahn, Youngjae Yu

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [block](<https://devfeed.tech/tags/block.md>), [channel](<https://devfeed.tech/tags/channel.md>), [core](<https://devfeed.tech/tags/core.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [layer](<https://devfeed.tech/tags/layer.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [models](<https://devfeed.tech/tags/models.md>), [pairing](<https://devfeed.tech/tags/pairing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [writer](<https://devfeed.tech/tags/writer.md>)

### AI overview

The authors measure reader-writer coupling in transformer MLP channels using the exact Jacobian of an MLP's residual-stream update. A channel's signed pairing coordinate comes from the inner product between its reader and writer terms. Across the measured blocks, strongly coupled channels form a small core that is nearly all opposed where initialization allows either sign equally: writers point against the directions their readers respond to. This pattern appears in small models trained from scratch and released checkpoints, emerges within a few percent of training, and strengthens thereafter. Most channels' contributions are taken up by readers in later blocks, while few appear in logits. Flipping opposed writers moves outputs toward common tokens; permuting a writer onto another reader leads the writer to regrow to fit that reader while the reader stays.

### Source excerpt

arXiv:2610.29340v1 Announce Type: new Abstract: A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.

## The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

DevFeed: [The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction](<https://devfeed.tech/articles/the-kv-surgeon-compensation-aware-offline-kv-cache-compaction-93575.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25933>)

Author: Moritz Wagner, Christophe Roux, Max Zimmer, Sebastian Pokutta

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [key-value-cache](<https://devfeed.tech/tags/key-value-cache.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-models](<https://devfeed.tech/tags/mixture-of-experts-models.md>), [models](<https://devfeed.tech/tags/models.md>), [offline](<https://devfeed.tech/tags/offline.md>)

### AI overview

KV-Surgeon compacts a key-value cache for repeated queries by accounting for how deleting one entry changes attention weights across the remaining entries. Using calibration data, it derives a quadratic surrogate with closed-form greedy removal and compensation decisions. The abstract reports improved cost-accuracy tradeoffs across QuALITY, LongHealth, and Long Code Arena for dense and mixture-of-experts models from 4B to 30B parameters. At 50x compaction, it improves question-answering accuracy by up to four points over attention matching at comparable cost; a faster variant also exceeds that method while using two-fifths of its cost.

### Source excerpt

arXiv:2610.25933v1 Announce Type: new Abstract: Offline compaction of the key--value cache (KV-cache) reduces the cost of repeatedly querying a fixed context by constructing a smaller cache once and reusing it subsequently. The central challenge of compaction is deciding which cache entries to delete and which to retain. Existing methods score each candidate for deletion on its own, with the rest of the cache being held fixed. However, softmax renormalization of the attention operation couples these decisions in a nonlinear manner: removing one entry changes the attention weights of every remaining one. In this work, we introduce KV-Surgeon, which uses calibration data to take interactions into account in a tractable way. We derive a quadratic surrogate that admits closed-form expressions for jointly making the optimal greedy removal decision as well as compensation within the remaining entries. On QuALITY, LongHealth, and Long Code Arena, and at every evaluated compaction ratio, KV-Surgeon substantially advances the cost--accuracy Pareto frontier of dense and mixture-of-experts models from 4B to 30B parameters. At $50\times$ compaction, KV-Surgeon improves question-answering accuracy by up to four points over attention matching (Zweiger et al., 2026), the state of the art, at comparable cost. Lastly, we introduce an even faster variant, which also exceeds the state of the art while requiring only two fifths of its cost.

## Certified Sparse Attention via Perturbed Retrieval

DevFeed: [Certified Sparse Attention via Perturbed Retrieval](<https://devfeed.tech/articles/certified-sparse-attention-via-perturbed-retrieval-93397.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.24991>)

Author: Runguo Li

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [error](<https://devfeed.tech/tags/error.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grouped-query-attention](<https://devfeed.tech/tags/grouped-query-attention.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [query](<https://devfeed.tech/tags/query.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scan](<https://devfeed.tech/tags/scan.md>), [score](<https://devfeed.tech/tags/score.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [values](<https://devfeed.tech/tags/values.md>)

### AI overview

Independent Exp(1) perturbations turn threshold retrieval into sampling keys in proportion to their attention weights. The method builds nested samples across geometric thresholds, estimates each output with a Horvitz-Thompson estimator, and stops each head when a simultaneous confidence bound meets the target error; fresh perturbations bound each certificate's failure probability during generation. The estimator reads O(ε⁻² log(1/δ)) values independent of context length, while its exact candidate scan uses 4-bit deterministic bounds shared across a grouped-query-attention group and still reads every key. In tests across three LLMs and ten settings at 16K-128K tokens, no certificate was violated. At equal keys read, its 90th-percentile error was 4.6-7.9x lower than the best of 15 baseline families; at equal bytes, baseline error was 0.6-1.6x its 90th-percentile error, though at least 4.9x its largest error. Task scores stayed within one point of dense attention, PG-19 perplexity within 0.1%, and CPU decoding was 1.1-1.4x faster at 32K-128K tokens on the same 16-bit cache.

### Source excerpt

arXiv:2610.24991v1 Announce Type: new Abstract: Sparse attention speeds up long-context decoding by reading few keys of the KV cache, but existing methods do not control the error of a query, and on real KV caches their errors have heavy tails. We show that perturbing the score of every key with an independent Exp(1) variable turns threshold retrieval into sampling keys with probability proportional to their attention weights. Lowering the threshold along a geometric grid gives nested samples, a Horvitz--Thompson estimate of the output at every level, and confidence bounds that hold simultaneously over all levels, so each head stops at the first level whose certificate meets the target error $\varepsilon$. With fresh perturbations at every decoding step, each certificate fails with probability at most $\delta$ during generation. At a suitable threshold, the estimator reads $O(\varepsilon^{-2}\log(1/\delta))$ values regardless of the context length, and no adaptive method reads fewer, up to constants. A 4-bit scan with deterministic bounds, shared across a grouped-query-attention group, generates the candidates exactly but reads every key. On three LLMs and ten settings at 16K--128K tokens, no certificate is violated, and at equal keys read our 90th-percentile error is 4.6--7.9$\times$ lower than that of the best of 15 baseline families. At equal bytes, the best baselines have 0.6--1.6 times our 90th-percentile error and at least 4.9 times our largest. End to end, our method stays within one point of dense attention on tasks and within 0.1\% in PG-19 perplexity, and the strongest baselines at equal bytes score as well. On CPU, decoding steps at 32K--128K tokens run 1.1--1.4$\times$ faster than dense attention on the same 16-bit cache.

## AI Company Lets Cops Pinpoint Specific Houses Based on a Single Interior Photo

DevFeed: [AI Company Lets Cops Pinpoint Specific Houses Based on a Single Interior Photo](<https://devfeed.tech/articles/ai-company-lets-cops-pinpoint-specific-houses-based-on-a-single-interior-photo-85314.md>)

Original publisher: [Read original article](<https://www.404media.co/ai-company-lets-cops-pinpoint-specific-houses-based-on-a-single-interior-photo/>)

Author: Joseph Cox

Published: 2026-10-12T13:00:05Z

Content type: news

Language: en

Sources: [404 Media](<https://devfeed.tech/sources/404-media.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [ARKTunnel](<https://devfeed.tech/topics/arktunnel.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [exploitation](<https://devfeed.tech/tags/exploitation.md>), [news](<https://devfeed.tech/tags/news.md>), [raven](<https://devfeed.tech/tags/raven.md>)

### AI overview

The article reports that Raven, an AI geolocation tool formerly called GeoSpy, can use interior photos to identify likely homes, rentals, or hotels by matching visual clues with property listings. It describes concerns about law enforcement use, data scraping, and privacy, and notes that a sheriff's office said the tool had not produced significant investigative breakthroughs or arrests.

### Source excerpt

Raven, previously called GeoSpy, lets cops figure out what specific Airbnb or Zillow property a photo was taken in.

## Tracing distinctive language in AI-written text

DevFeed: [Tracing distinctive language in AI-written text](<https://devfeed.tech/articles/tracing-distinctive-language-in-ai-written-text-82644.md>)

Original publisher: [Read original article](<https://allenai.org/blog/infinigram-books>)

Published: 2026-07-31T08:00:00Z

Content type: news

Language: en

Sources: [Ai2 Blog](<https://devfeed.tech/sources/ai2-blog.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [watermarking](<https://devfeed.tech/topics/watermarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [detection](<https://devfeed.tech/tags/detection.md>), [published](<https://devfeed.tech/tags/published.md>)

### AI overview

Researchers used Ai2's infini-gram engine to compare rare phrases in self-published books with and without substantial detected AI text. In the top-selling books studied, those with substantial detected AI text contained more rare expressions associated with previously published books. The researchers describe phrase overlap as evidence that can support detector scores, while noting that it does not by itself establish where a phrase originated.

### Source excerpt

Stony Brook researchers used our infini-gram engine to trace distinctive phrases in AI-generated writing back to existing sources, finding that top-selling self-published books on Amazon with substantial detected AI text overlap more heavily with rare language from previously published works.

## A failed experiment: Infini-Attention, and why we should keep trying?

DevFeed: [A failed experiment: Infini-Attention, and why we should keep trying?](<https://devfeed.tech/articles/a-failed-experiment-infini-attention-and-why-we-should-keep-trying-7286.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/infini-attention>)

Author: Phuc Nguyen; Leandro von Werra; Thomas Wolf

Published: 2024-08-14T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>)

Tags: [complexity](<https://devfeed.tech/tags/complexity.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infini-attention](<https://devfeed.tech/tags/infini-attention.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory-compression](<https://devfeed.tech/tags/memory-compression.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>)

### AI overview

This article examines Infini-attention as an alternative approach for extending language-model context lengths. It explains the quadratic memory cost of standard self-attention, the limitations of Flash Attention and Ring Attention, and the resource demands of training models with million-token contexts.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.