# A "feature ODE" describing the learning behavior of shallow MLPs on simple functions

DevFeed: [A "feature ODE" describing the learning behavior of shallow MLPs on simple functions](<https://devfeed.tech/articles/a-feature-ode-describing-the-learning-behavior-of-shallow-mlps-on-simple-functions-93525.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.25652>)

Author: Joseph Turnbull, Berkan Ottlik, Jamie Simon

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feature](<https://devfeed.tech/tags/feature.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [simple](<https://devfeed.tech/tags/simple.md>), [training](<https://devfeed.tech/tags/training.md>), [two-layer](<https://devfeed.tech/tags/two-layer.md>)

## AI overview

The proposed "feature ODE" roughly matches the loss trajectories of two-layer MLPs trained by gradient flow on anisotropic Gaussian data to learn low-order Hermite polynomials and staircase functions. This is notable because the ODE substantially simplifies the dynamics and is not derived through a rigorous chain of approximations. The authors discuss the surprising agreement and speculate about its possible reasons and uses; the abstract does not establish a rigorous explanation.

## Source excerpt

arXiv:2610.25652v1 Announce Type: new Abstract: We study the gradient flow training dynamics of two-layer multi-layer perceptrons (MLPs) on anisotropic Gaussian data. Even in this simple setting, solving the dynamics in full is notoriously difficult, so we try something different. We propose a "feature ODE" that might be intuitively expected to describe the learning dynamics of a shallow MLP. We find that MLPs trained to learn simple target functions (low-order Hermite polynomials and staircase functions) actually roughly follow the loss trajectories from this feature ODE. This agreement is a surprise because of the size of the simplification and the fact that it does not come from a rigorous chain of approximations. We discuss this surprise and speculate on its reasons and uses.