# Towards Training Transformers as a Universal Computer

DevFeed: [Towards Training Transformers as a Universal Computer](<https://devfeed.tech/articles/towards-training-transformers-as-a-universal-computer-107888.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.25166>)

Author: Ruize Xu, Chenxiao Yang, Yanhong Li, David McAllester

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Transformers](<https://devfeed.tech/topics/transformers.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Software](<https://devfeed.tech/topics/software.md>)

Tags: [addition](<https://devfeed.tech/tags/addition.md>), [bounded-context](<https://devfeed.tech/tags/bounded-context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [execution](<https://devfeed.tech/tags/execution.md>), [language](<https://devfeed.tech/tags/language.md>), [out-of-distribution-generalization](<https://devfeed.tech/tags/out-of-distribution-generalization.md>), [pencil](<https://devfeed.tech/tags/pencil.md>), [programming-language](<https://devfeed.tech/tags/programming-language.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [verification](<https://devfeed.tech/tags/verification.md>)

## AI overview

A small transformer trained on randomly generated MicroPy programs learned to predict their small-step execution when given procedure definitions and an expression to evaluate. PENCIL scaffolding supports space-efficient execution within a bounded context window. The model reportedly generalized to held-out human-written programs for bit copying and flipping, binary addition and multiplication, and SAT verification and solving, and evaluated programs from a substantially different distribution than its training data. Because MicroPy is computationally universal, the results provide empirical evidence that a standard transformer paired with PENCIL can act as a universal computer; they do not establish unbounded execution.

## Source excerpt

arXiv:2604.25166v2 Announce Type: replace Abstract: We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language. Given procedure definitions together with an expression to evaluate, the transformer predicts small-step execution using PENCIL scaffolding for space-efficient execution within a bounded context window. After training on randomly generated, semantically arbitrary MicroPy programs, the learned transformer generalizes to held-out human-written programs including bit copying and flipping, binary addition and multiplication, and SAT verification and solving. The trained model exhibits out-of-distribution generalization: it correctly evaluates programs drawn from a substantially different distribution than the one used to generate its training data. Since MicroPy can express any computation, our results provide empirical evidence that a standard transformer can be trained to act as a universal computer when paired with PENCIL scaffolding. Project page: https://www.micropy.org