# First-Token Attraction in Mamba Dynamics

DevFeed: [First-Token Attraction in Mamba Dynamics](<https://devfeed.tech/articles/first-token-attraction-in-mamba-dynamics-120306.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08303>)

Author: Trinh T. Nguyen, Minh-Khoi Nguyen-Nhat, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. Vo

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mamba](<https://devfeed.tech/topics/mamba.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [clustering](<https://devfeed.tech/topics/clustering.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [attention-sink](<https://devfeed.tech/tags/attention-sink.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deep](<https://devfeed.tech/tags/deep.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [falcon](<https://devfeed.tech/tags/falcon.md>), [invariant](<https://devfeed.tech/tags/invariant.md>), [layers](<https://devfeed.tech/tags/layers.md>), [limit](<https://devfeed.tech/tags/limit.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [models](<https://devfeed.tech/tags/models.md>), [points](<https://devfeed.tech/tags/points.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [state-space-models](<https://devfeed.tech/tags/state-space-models.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

## AI overview

The abstract reports that, in deep Mamba models, the first token's limiting direction attracts later tokens, producing clustering and attention-sink behavior. To analyze matrix-valued interactions lacking positivity or definiteness guarantees, the authors introduce a framework using a high-dimensional invariant box and a sharp exit-time argument; they say it characterizes token-trajectory limit points and proves explicit exponential convergence rates. They report that the mechanism also extends to state-space models, causal attention, and classical time-series models. Experiments on Falcon-Mamba-7B are said to show increasing concentration on the first token with depth, dominating in later layers.

## Source excerpt

arXiv:2611.08303v1 Announce Type: new Abstract: We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba dynamics involve full matrix-valued interaction coefficients with no positivity or definiteness guarantees. This structural difference makes Mamba dynamics substantially harder to analyze and prevents the direct application of existing techniques for attention-based models. To overcome this challenge, we introduce a new analytical framework based on a high-dimensional invariant box and a sharp exit-time argument, which together recover an effective form of sign-definiteness. Using this framework, we characterize the possible limit points of token trajectories and prove explicit exponential convergence rates. Our analysis reveals that the first token has privileged dynamics: its limiting direction attracts subsequent tokens, inducing token clustering and attention-sink phenomena in deep Mamba models. We further show that this attraction mechanism extends to broader sequence dynamics, including state-space models, causal attention, and classical time-series models. Experiments on Falcon-Mamba-7B validate our predictions, showing that concentration on the first token increases with depth and dominates in later layers.