# Generative sequence modeling for infinite memory processes via predictive states

DevFeed: [Generative sequence modeling for infinite memory processes via predictive states](<https://devfeed.tech/articles/generative-sequence-modeling-for-infinite-memory-processes-via-predictive-states-135720.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.38524>)

Author: Michael Wieck-Sosa, Cosma Rohilla Shalizi

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [World models](<https://devfeed.tech/topics/world-models.md>)

Tags: [complexity](<https://devfeed.tech/tags/complexity.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [curse-of-dimensionality](<https://devfeed.tech/tags/curse-of-dimensionality.md>), [data](<https://devfeed.tech/tags/data.md>), [deep](<https://devfeed.tech/tags/deep.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [faster](<https://devfeed.tech/tags/faster.md>), [feature](<https://devfeed.tech/tags/feature.md>), [future](<https://devfeed.tech/tags/future.md>), [generative](<https://devfeed.tech/tags/generative.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [history](<https://devfeed.tech/tags/history.md>), [infinite](<https://devfeed.tech/tags/infinite.md>), [memory](<https://devfeed.tech/tags/memory.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [processes](<https://devfeed.tech/tags/processes.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

## AI overview

The proposed estimator predicts the next-step distribution of a multivariate process by learning predictive states that compress past history into a statistic sufficient for predicting the future. Its statistical complexity depends on the predictive-state dimension; the authors show that a p-dimensional predictive-state space can be represented in 2p+1 dimensions, suggesting faster convergence when this state is low-dimensional. They give convergence guarantees for a deep-neural-network version and report evaluations on synthetic data and real C. elegans data. The abstract frames this approach as addressing long-range nonlinear interactions that finite-memory, sparse, or additive assumptions may miss, while avoiding the curse of dimensionality remains challenging without such assumptions.

## Source excerpt

arXiv:2609.38524v3 Announce Type: replace-cross Abstract: We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimator based on learning representations of the \emph{predictive states} of a stochastic process. We show that the statistical complexity of the estimation problem is determined by the dimension of this predictive state representation. Furthermore, we show that a $p$-dimensional predictive state space can be represented in a $2p+1$-dimensional feature space. Our results imply that faster convergence rates are achievable when the predictive state space is lower-dimensional, i.e., when the past history can be compressed into a lower-dimensional statistic that is sufficient for predicting the future. We establish theoretical convergence rate guarantees for an instantiation of our method based on deep neural network estimators. We implement our method in PyTorch and evaluate its performance on synthetic data and real \emph{C. elegans} data.