# What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

DevFeed: [What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language](<https://devfeed.tech/articles/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of-language-140119.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23093>)

Author: Aritra Das, Francesco Cagnetta, Darshil Doshi, Victor V. Albert, Maissam Barkeshli, Andrey Gromov

Published: 2026-11-18T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [composition](<https://devfeed.tech/tags/composition.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [conditional-probability](<https://devfeed.tech/tags/conditional-probability.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [gram](<https://devfeed.tech/tags/gram.md>), [language](<https://devfeed.tech/tags/language.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [routing](<https://devfeed.tech/tags/routing.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tensors](<https://devfeed.tech/tags/tensors.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

## AI overview

Tree-grams are autoregressive n-gram models that compose order-2 and order-3 tensors to form conditional probabilities. Their context window is controlled by n and their capacity by rank d, providing interpretable controls over each. The authors report that tree-grams outperform existing n-gram baselines and remain competitive with deep transformers. As generative models, they make narrower-model hyperparameter transfer under μP more robust when tuning on simpler tree-gram-generated data with moderate n and d. The abstract also reports that entropy and correlation range shape post-fine-tuning performance gaps, and that routing across candidate topologies becomes sparse enough to prune up to 95% without significant performance change.

## Source excerpt

arXiv:2611.23093v1 Announce Type: cross Abstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.