# Next-Latent Prediction Transformers Learn Compact World Models

DevFeed: [Next-Latent Prediction Transformers Learn Compact World Models](<https://devfeed.tech/articles/next-latent-prediction-transformers-learn-compact-world-models-132735.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2511.05963>)

Author: Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [lookahead](<https://devfeed.tech/tags/lookahead.md>), [planning](<https://devfeed.tech/tags/planning.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

## AI overview

Next-Latent Prediction (NextLat) adds self-supervised predictions of a transformer's next latent state, conditioned on the next token, to standard next-token training. The authors argue this encourages compact belief states and coherent transition dynamics, adding a recurrent bias without changing the architecture, parallel training efficiency, or inference. They report gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning across world-modeling, reasoning, planning, and language-modeling benchmarks. NextLat also supports variable-length self-speculative decoding, with reported language-modeling inference acceleration of up to 3.3x. These are abstract-level claims; the supplied text does not give benchmark details or experimental qualifications.

## Source excerpt

arXiv:2511.05963v5 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction (NextLat), which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next token. Theoretically, we show that these latents converge towards belief states, compressed information about the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged. NextLat effectively encourages transformers to form compact internal world models with coherent belief states and transition dynamics -- crucial properties not guaranteed by standard next-token prediction alone. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3x in language modeling. NextLat offers a simple yet effective paradigm for learning compact, predictive representations in transformers that generalize better. Our code is available at https://github.com/JaydenTeoh/NextLat.