# Implicit Bias in State Space Models and Linear Autoregressive Training

DevFeed: [Implicit Bias in State Space Models and Linear Autoregressive Training](<https://devfeed.tech/articles/implicit-bias-in-state-space-models-and-linear-autoregressive-training-125152.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10237>)

Author: Gal Katzhendler, Gilad Yehudai, Gal Vardi

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mamba](<https://devfeed.tech/topics/mamba.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [bias](<https://devfeed.tech/tags/bias.md>), [classification](<https://devfeed.tech/tags/classification.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [linear](<https://devfeed.tech/tags/linear.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [models](<https://devfeed.tech/tags/models.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [state-space-models](<https://devfeed.tech/tags/state-space-models.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

The paper characterizes the implicit bias of gradient descent for linear state space and autoregressive models with shared parameters. Under interpolation, directional-convergence, and trajectory-dependent effective-degree assumptions, gradient descent on exponential loss converges in direction to a KKT point of an effective-degree max-margin problem. Only examples with the smallest effective degree impose hard margin constraints; examples with faster-growing margins become asymptotically inactive apart from feasibility. The authors identify a possible length bias: longer sequences or additional autoregressive steps can raise an example's effective degree, leaving examples with the smallest degree to dominate the limiting classifier. This first-order result shows how parameter sharing and non-homogeneity change classical homogeneous implicit-bias characterizations.

## Source excerpt

arXiv:2611.10237v1 Announce Type: new Abstract: Linear State Space Models (SSMs) and linear autoregressive sequence models reuse the same parameters across a sample-dependent number of steps. As a result, different examples impose constraints through different predictors, even though all predictors share the same weights. We study this phenomenon through a general model of multiple non-homogeneous polynomial predictors with shared parameters, in a classification setting. Under standard interpolation and directional-convergence assumptions, together with a trajectory-dependent effective-degree condition, we show that gradient descent on the exponential loss converges in direction to a KKT point of an effective-degree max-margin problem. In this problem, only examples with minimal effective degree impose hard margin constraints; examples with faster-growing margins are asymptotically inactive except for feasibility. Our result provides a first-order characterization of the implicit bias in linear SSMs and linear autoregressive models. It exposes a mechanism of length bias: a longer input sequence or additional autoregressive steps may increase the example's effective degree, and the examples with the smallest effective degree dominate the limiting classifier. Our result highlights how non-homogeneity and parameter sharing alter classical homogeneous implicit-bias results.